Mounting USB devices in WSL-2

My goal was to deduplicate all the images and videos I have in my large backup USB hard drive. Obviously this was a task for Linux but the only Linux I have ready access to these days is WSL-2 running under Windows 11.

By default USB devices connected on the Windows side are not accessible from WSL-2. The solution is laid out in this how-to from Microsoft.

Setup

Prequisite: WSL needs to have kernel which is USBIP capable. This can be achieved by updating WSL in Powershell: wsl --update.

Windows side

Step 1: Install the (open-source) usbipd-win utility (msi) from the project's releases page. Verify the installation by launching Windows Powershell and running usbipd list.

WSL side

We now need to compile the Kernel to enable CONFIG_USB_STORAGE.

Step 1: Figure out the current kernel version. In Powershell run: wsl --version. In our example the kernel version is 5.15.153.1-2.

Step 2: Clone the WSL kernel source code for the version installed: git clone --depth 1 --branch linux-msft-wsl-5.15.153.1 https://github.com/microsoft/WSL2-Linux-Kernel.git

Step 3: Install build tools: sudo apt-get install git bc build-essential flex bison libssl-dev libelf-dev dwarves

Step 4: Install usbip tools on the WSL side: sudo apt-get install usbutils hwdata

Step 5: 

cd WSL2-Linux-Kernel

cp Microsoft/config-wsl .config

scripts/config --enable USB_STORAGE

make oldconfig

The last command will prompt for a bunch of information. Answer y to everything except "verbose debug" (and possibly Rio karma music player).

make -j$(nproc)

cp arch/x86/boot/bzImage /c/Users/<windows-user>/

Step 6: Modify wsl config to use this compiled kernel by modifying the /c/Users/<windows-user>/.wslconfig file and pointing the kernel property to C:\\Users\\<windows-user>\\bzImage

Step 7: Restart WSL (in Powershell run: wsl --shutdown and then launch WSL again)

Sources:

• https://renenyffenegger.ch/notes/Windows/Subsystem-for-Linux/index#wsl-compile-kernel
• https://renenyffenegger.ch/notes/Windows/Subsystem-for-Linux/index

Mounting USB device in WSL

Step 1: In Powershell run usbipd list and take a look at the devices that are already attached. We will rerun this command and look at the change to discover our device.

Now attach your USB device and eject it (while keeping it plugged in). Attachment will fail if Windows is accessing the USB device.

Step 2: In Powershell run usbipd list and determine the busid (BUS ID) of the attached device. It is of the form d-d (where d is an integer). In our example the busid will be 2-6.

We will now bind the USB devise to WSL (make sure that the WSL VM is up, opening a WSL terminal should suffice). 

Step 3: In an Admin Powershell: usbipd bind --busid 2-6. To verify run usbipd list and check that the STATE for your device now says Shared.

Step 4: In (a normal) Powershell: usbipd attach --wsl --busid 2-6. Verify by running usbipd list and checking that the STATE for your device now says Attached.

Step 5: In WSL run lsusb to see the USB device showing up. Then run dmesg and look at the last bit of output. We are looking for usb, scsi, and sd output, in particular any mention of a disk partition e.g. sdd1.

Step 6: Mount the partition: mount /dev/sdd1 /mnt/usb

Unmounting USB device in WSL

Step 1: Unmount partition in WSL: umount /mnt/usb

Step 2: Unattach device in Powershell: usbipd unattach --busid 2-6

Step 3: Unbind device in Powershell: usbipd unbind --basuid 2-6

Installing, configuring and using tin to read Usenet articles in Linux (WSL-2)

 I recently learned about Gemini and how its development is being tracked on Usenet. This got me interested in accessing Usenet.

Registering with Usenet Provider

The first step is to register with a Usenet provider. Since I am primarily interested in the comp.infosystems.gemini group I can use a free provider which only gives text access. I chose to use https://www.eternal-september.org/. Registration was extremely straight forward and I received credentials via email.

Installing tin (Usenet reader)

I looked around for a Linux based Usenet reader and came across tin. I am currently using a Windows laptop with WSL-2 installed running Ubuntu 20.04. I installed tin using:

sudo apt-get install tin

Authenticating with tin

I created a ~/.newsauth file readable only by myself (chmod go-r ~/.newsauth) and placed my credentials inside it in the recommended format:

news.eternal-september.org <password> <username>

All of this information came from the registration email I got from eternal-septemeber.

Configuring tin

To use neovim as the editor of choice when replying to articles I set export EDITOR='nvim' in my ~/.zshrc. I also set TIN_HOMEDIR="$HOME/.tin" to consolidate the various configuration files.

To correctly identify myself when posting articles (replies) I modified ~/.tin/tinrc and set mail_address="Abid H. Mujtaba" <abid.mujtaba@protonmail.com>

Running tin

Launch via: tin -r -g news.eternal-september.org -A comp.infosystems.gemini

-r tells tin to fetch from articles from the remote server (rather than local). -g is used to specify which remote server to use (eternal-september in my case), and -A tells tin to authenticate with the remote server (using the credentials in ~/.newsauth, prompts for creds if that file doesn't exist).

One can set NNTPSERVER="news.eternal-september.org" to avoid having to specify -g. Similarly one can (initially) place the line comp.infosystems.gemini: (note the necessary colon at the end) in ~/.tin/.newsrc (extra info will be appended after the colon automatically as you interact with the usenet group). With these changes one can simply issue tin -r -A.

Using tin

To get started, select a thread (of articles). To read the next article in the thread click n (p for the previous article). Use j and k to move down and up inside an article. Can also enter the article number (within the thread) to jump to it.

To post in response to an article use f (follow-up). This is different then replying (r) which sends an email to the author (using sendmail).

 

Compiling Latex with latexmk with tikz externalization and pre-compiled preamble (which includes tikz)

Compilation of LaTeX projects that contain tikz images can be sped up by an astronomical amount by properly separating the static and dynamic preamble and issuing the right commands.

A minimum-working-example that I wrote that demonstrates this in great detail can be found at: https://github.com/abid-mujtaba/mwe-latexmk-tikz-preamble

Ubuntu - Copy large number of files easily to Android Phone

Situation:

The MTP protocol for transferring files to and from an Android Phone is utter garbage, especially when transferring a large number of files (like your Music collection for the first time - on my Google Pixel). The connection inevitably fails in the middle of the transfer, which is slow to boot.

Solution:

USB tether your phone to your computer (read http://abidmujtaba.blogspot.com/2017/07/ubuntu-usb-tethering-with-google-pixel.html if you get stuck). You don't even need internet pass-through, just an IP Address (so you can use sudo dhclient usb0 instead of sudo dhcpcd usb0 if you are using the terminal).

Next follow http://abidmujtaba.blogspot.com/2017/07/ubuntu-ssh-in-to-android-phone-google.html to set up an ssh connection. Once you have that you are home free.

I use lftp over ssh to copy all the files I want. I even have a simple lftp bookmark set up:

pixel sftp://abid:xx@pixel

where my ssh config section is titled "pixel" and the password "xx" is just a red-herring since ssh is configured to use an RSA key for authentication. This bookmark allows me to access my phone's files using a simple lftp pixel.

Ubuntu - SSH in to Android Phone (Google Pixel)

The ability to SSH in to your (unrooted) Android Phone is a very powerful one to have, and it is surprisingly simple.

1. Install the "SimpleSSHD" app on your phone (form the Google Play Store).
2. Copy your PUBLIC ssh key (form your computer) to the "/sdcard/ssh" (default) folder on your phone and rename it authorized_keys.
3. Run the "SimpleSSHD" app and click start (uses port 2222 be default since it runs in user space). The app will tell you the IP address of the phone.
4. SSH in to your phone by issuing:

   ssh -o StrictHostKeyChecking=no -p 2222 -i <private key location> <phone ip address>

   You will need to issue StrictHostKeyChecking=no the first time only.
5. To avoid having to type all of this out every time you can add a section to the ssh config file (~/.ssh/config).

   Host pixel
   
       Hostname 192.168.42.129
       Port 2222
       IdentityFile ~/.ssh/pixel
   

   and simply run ssh pixel to connect.

Nore: This technique works with USB tethering as well. SimpleSSHD will show both the WiFi and the USB Tethering IP Addresses and you can use the latter for super-fast ssh.

Ubuntu - USB Tethering with the Google Pixel

Situation:

When I connected my Google Pixel to my Ubuntu (14.04) laptop with the USB cable and selected the option to USB tether on the phone the connection would fail. The Network Manager applet would show a greyed out option "Ethernet Network (Google Pixel)" with the status "disconnected" underneath it, also greyed out.

Interestingly running "ifconfig" revealed that a "usb0" section was present but no IP address had been assigned.

usb0      Link encap:Ethernet  HWaddr 32:a8:e6:6c:fe:9d  
          inet6 addr: fe80::30a8:e6ff:fe6c:fe9d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:560 (560.0 B)  TX bytes:1802 (1.8 KB)

This clue led to a long trawl across the internet that led to a surprisingly simple solution.

Solution:

Install the "dhcpcd" package.

sudo apt-get install dhcpcd

Now after connecting the Google Pixel simply run:

sudo dhcpcd usb0

And confirm by running:

ifconfig usb0
route -n
ping -c3 www.google.com

Compiling Latex with scons with tikz externalization and pre-compiled preamble (which includes tikz)

Edit: I have created another MWE that uses latexmk instead of scons (which I found to be using more resources and not recompiling correctly when citations were added): http://abidmujtaba.blogspot.com/2017/10/compiling-latex-with-latexmk-with-tikz.html

Compilation of LaTeX projects that contain tikz images can be sped up by an astronomical amount by properly separating the static and dynamic preamble and issuing the right commands.

A minimum-working-example that I wrote that demonstrates this in great detail can be found at: https://github.com/abid-mujtaba/mwe-scons-tikz-preamble

Ubuntu - Connecting to a PPTP VPN server using the terminal

Situation:

I was away from Islamabad and needed to download some scientific papers. This required me to connect to my University's VPN which was of the PPTP variety. I tried using the Network Manager applet in Ubuntu 14.04 which would successfully connect but the moment I would open a page on Chrome the VPN connection would fail.


Solution:

Connect to the VPN from the terminal using instructions that were written for Ubuntu 7.04 (https://wiki.ubuntu.com/VPN) and were supposed to have become outdated 7 years ago. But what is in the command line can never die.

Based on help from https://superuser.com/a/459906/306711 and http://www.wikihow.com/Add-or-Change-the-Default-Gateway-in-Linux with a little tweaking I was able to successfully connect to the VPN in a stable fashion.


Details:

1. Install pptp package: sudo apt-get install pptp-linux


2. Create a peers file. I named mine 'comsats' so I created: /etc/ppp/peers/comsats with the following content

       pty "pptp <vpn server ip address> --nolaunchpppd"
       debug
       nodetach
       logfd 2
       noproxyarp
       ipparam comsats
       remotename comsats
       name <vpn username>
       require-mppe-128
       nobsdcomp
       nodeflate
       lock
       noauth
       refuse-eap
       refuse-chap
   

   This file needs to be executable (sudo chmod +x /etc/ppp/peers/comsats). Note the 'remotename' and the 'ipparam' which are set to 'comsats'. These will be used to activate the connection and to run automated scripts when the connection is setup and torn down.


3. Add your VPN credentials to /etc/ppp/chap-secrets in the following format:

       * <vpn username> * <vpn password>

   At this point you can check your vpn connection by executing:

       sudo pon comsats

   For debugging the connection look at /var/log/syslog. At this stage your internet will NOT be forwarded over the vpn because the routes haven't been set up for it.
   
   To close the connection Ctrl+C is supposed to work but it didn't for me so I had to use ps -A | grep ppp to find the pid of the connection process and then sudo kill -KILL <pid> to kill it.


4. To forward your internet packets over the VPN you must first find the ip addresses of the gateways for your internet interface (wlan0 in my case) and the vpn interface (ppp0). Simply use ipconfig. In my case my wifi gateway was 192.168.0.1 and the vpn gateway was 192.168.3.1. To take a look at the current routes issue the route command, it will show you the current default gateway.
   The first step is the delete the current default gateway:

       sudo route delete gw 192.168.0.1 wlan0

   Note the use of my wlan0 gateway ip address (192.168.0.1).
   Now switch to using the ppp0 gateway as the default so that all outgoing internet traffic is sent via the vpn:

       sudo route add gw 192.168.3.1 ppp0

   You can confirm the switch to using the VPN by looking at your external IP address (I simply navigate to https://www.icanhazip.com and a whois to confirm).


5. After closing the vpn connection you will have to revert the changes to the routes to get your normal internet access back (the Network Manager applet does all of this automatically, when it works) as follows:

       sudo route delete gw 192.168.3.1 ppp0
       sudo add gw 192.168.0.1 wlan0



6. To automate the route changes one can add files to the /etc/ppp/ip-up.d and /etc/ppp/ip-down.d folders that are automatically run when the connection is setup and torn down respectively.
   Create and populate /etc/ppp/ip-up.d/comsats:

       #!/bin/sh
   
       [ "$PPP_IPPARAM" = "comsats" ] || exit 0
       route delete default gw 192.168.0.1 wlan0
       route add default gw 192.168.3.1 ppp0
   

   Note how we use the $PPP_IPPARAM to match against the same we defined in /etc/ppp/peers/comsats to make the code run only when this connection is set up.
   Similarly create and populate /etc/ppp/ip-down.d/comsats:

       #!/bin/sh
   
       [ "$PPP_IPPARAM" = "comsats" ] || exit 0
       route delete default gw 192.168.3.1 ppp0
       route add default gw 192.168.0.1 wlan0
   

   Finally comment out nodetach in /etc/ppp/peers/comsats to run the connection as a deamon and use sudo poff comsats to close the connection when you need to. This connection is not as stable as I would like it (usable however) so I use tail -F /var/log/syslog | ccze -A to keep an eye on the connection while I am using it, restarting it as needed.

Highlighted (syntax) code snippets in Blogger

My blogs being technical in nature often contain snippets of code. The old technique I was using for correctly highlighting them no longer works because the library I was using has gone the way of the dinosaurs. Thankfully, however, there is a much better (and slightly easier) way to achieve this using the highlight.js library (https://highlightjs.org/).


Procedure:

1. In your main Blogger page, from the pane on the left (which contains 'Posts', 'Stats', etc.) click on 'Theme'.
2. In the frame that reloads on the right first choose 'Backup / Restore' and then 'Download theme' to save a backup of the unmodified theme before we mess with it.
3. Press 'Close' to close the dialog/pop-up window and choose 'Edit HTML'.
4. Before the </head> tag paste the following lines:

   
   <link href='//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.8.0/styles/agate.min.css' rel='stylesheet'/>
   <script src='//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.8.0/highlight.min.js'/>
   <script>hljs.initHighlightingOnLoad();</script>
   

5. Choose 'Save Theme'

Note:   I chose to use the 'agate' theme (CSS), there are many others. You can take a look at the available options at https://highlightjs.org/static/demo/. To choose another one simply replace "agate" in the link with the theme you like.

Similarly, 'highlight.min.js' only supports a core of 35 common languages. To highlight other languages add additional scripts you can find at https://cdnjs.com/libraries/highlight.js/



Example:

Now to place a code snippet in your blog simply switch to "HTML" editing (instead of "Compose") and place the snippet inside a <pre><code> </code></pre> block or simply inside <code> </code> if quoting code inline. You can specify the language inside the "class" element of the "code" block (visit the highlight.js website for more details).

To create the following highlighted python snippet:

    def factorial(n):
        if n == 0:
            return 1
        else:
            return n * factorial(n - 1)

I used the following code in HTML mode:

    <pre><code class="python">    def factorial(n):
        if n == 0:
            return 1
        else:
            return n * factorial(n - 1)
    </code></pre>

Simple and elegant. Enjoy.

Note: To place HTML itself inside a highlighted snippet (very meta indeed, I needed it for the last section of code) you will have to escape all of the HTML. I used https://www.freeformatter.com/html-escape.html.

Linux - Switch display from terminal (virtual and emulated) using xrandr

Situation: I had my laptop (running Ubuntu) connected to my TV via an HDMI cable with the display set to TV only. I then unplugged the HDMI cable without switching the display to Internal (using the "Displays" system settings). The display is supposed to switch automatically but on my machine this sometimes fails to happen leaving me with a running X session which I can't see.

However by pressing Ctrl+Alt+F2 (through F6) I am able to switch to a virtual terminal which shows up on the internal screen just fine.

Solution: Use xrandr to switch the display. But this requires two little tricks because xrandr is an X command and it normally fails on virtual terminals.

sleep 5 && xrandr -d :0 --output LVDS1 --auto

and quickly switch to the terminal running X (Ctrl+Alt+F7 on Ubuntu) before the sleep timer runs out.

Explanation: By sleeping for 5 seconds we allow ourselves time to switch to the virtual terminal running X so that when the xrandr command is issued the X Server is awake to receive it.

Since we are running xrandr from the virtual terminal we have to explicitly specify which display to target. The default display is ":0" meaning the first (zero-indexed) display on the local machine (for remote machine its IP address would precede the ':').

Finally --output LVDS1 --auto tells xrandr to switch to the internal display with default settings.

Ubuntu - Create a wifi hotspot (access point accessible by Android devices) whose traffic is forwarded through a SOCKS proxy

The objective is to take my ubuntu laptop connected to the internet via ethernet and create a wifi hotspot (access point) that will allow an android device to connect. The hotspot will provide internet access to all connected devices.

The crucial feature is that all traffic from devices connected to the hotspot must pass through a SOCKS proxy. This means that all requests and connections from these devices seem to come from the external end of the SOCKS connection.

Although the internet is full of advice on this there is very little that works. Surprisingly the solution is actually quite simple.

The first step is to create the hotspot. On ubuntu the network-manager takes care of this with a simple modification. The steps are detailed in http://ubuntuhandbook.org/index.php/2014/09/3-ways-create-wifi-hotspot-ubuntu/. Basically create a new infrastructure mode wifi network using the network-manager and then edit a single configuration file (in /etc/NetworkManager/system-connections/) to change the mode to ap.

Once this network is enabled using the network-manager, your devices (including those running Android) will be able to connect the network and receive internet through the ethernet connection on your computer.

Step two is to create a SOCKS tunnel using ssh -D. This can be achieved using a single command on the terminal or by adding an entry in ~/.ssh/config for example:

Host <name>

    User <user>
    Hostname <host address>
    IdentityFile <full path to ssh key file>
    DynamicForward <port>
    ServerAliveInterval 60

With this configuration issuing the command ssh <name> will establish a connection the remote machine while the DynamicForward instruction will create a SOCKS5 tunnel such that any packets sent to port number <port> will be sent through the ssh tunnel. The packets will appear to come from the remote machine itself.

The SOCKS tunnel can be tested using a browser on the computer (not the hotspot yet) by setting it to use a SOCKS proxy. Point the browser at www.icanhazip.com and/or www.whoismyisp.org to check that packets are being sent over the tunnel.

The ssh socks port can interface with a browser (the latter being intelligent about it) but it won't handle raw ip traffic from the hotspot. We need a proxy that can take the traffic and send it via the ssh socks port. Enter redsocks a lightweight socks proxy. Unlike squid or polipo it is NOT a proxy server, that is, it cannot be used directly by a browser. redsocks connects to the socks port upstream and provides an incoming port where all incident traffic is sent out via the socks tunnel. Traffic is redirected to the redsocks incoming port using iptables.

Once redsocks is installed (sudo apt-get install redsocks) modify /etc/redsocks.conf. Focus on the redsocks {} section. Change local_ip to 0.0.0.0 (don't know if this is strictly necessary). local_port specifies the incoming port. I chose 12345.

Set port to the socks port set in the ssh config file and set type to socks5. Then restart redsocks (sudo service redsocks restart).

With redsocks set up we are now at the final stage and ready to redirect traffic from the wifi hotspot to through the socks tunnel. To that end we need the ip address block assigned to the hotspot. Since my hotspot works on the wlan0 interface I simply ran ifconfig wlan0 (or ip addr) to discover the address block which turned out to be 10.42.0.1/24, that is, address 10.42.0.1 and netmask 255.255.255.0.

When the network-manager creates the wifi hotspot (using the technique defined above) it creates a number of iptables rules to take care of the traffic forwarding. Since we need all traffic forwarded to the redsocks incoming port (12345) we simply delete all these rules and provided just one of our own.

    sudo iptables -F
    sudo iptables -t nat -F
    sudo iptables -t nat -A PREROUTING -s 10.42.0.0/24 -p tcp -j REDIRECT --to-ports 12345

Basically we are instructing iptables to take all packets that come from the 10.42.0.0/24 subnet and redirect it to the incoming port of redsocks which sends it over the socks ssh tunnel.

The final step is to enable ip packet forwarding in the kernel using

sudo sysctl -w net.ipv4.ip_forward=1

And we are set. On your hotspot connected device (Android phone for example) navigate to www.icanhazip.com and www.whoismyisp.org to confirm.

DIY mini-UPS for charging phones or running wifi router or Raspberry Pi

Currently Pakistan is going through an energy crisis. Consequently I am faced with frequent hour long power outages. I have a largish UPS for running an assortment of electrical appliances but I am always vary of using it to power DC electronics.

This is where this DIY mini-UPS comes in handy. It runs off a 12V motorcycle battery and is charged using a standard 12V DC power adapter. It is capable of powering either a standard wireless router (that requires 12V DC), a Raspberry Pi or charging a phone.

The switching from line voltage to battery voltage is fast enough that the wireless router doesn't stutter and the Raspberry Pi keeps running without a hitch.

I followed the instructions from this excellent tutorial which also gives a wonderful explanation for how the circuit works. I did make a few modifications of my own, simplifying the circuit to fit my needs. In particular I connected the DC output directly to the input because the voltage drop across both the 1N4007 diode and the TIP 127 was unacceptable when running off of the battery. The modified circuit is:

(the circuit diagram was drawn using circuitikz package in LaTeX. Here is the how-to)

And here is the end-product, shifted to a perf-board after being tested on a bread-board:
<insert image here>

Drawing Circuit diagrams in LaTeX

Those of you who are familiar with my interests would know that I am both an electronics hobbyist and a huge fan of LaTeX. Therefore I am thrilled with my recent discovery of the ability to draw publication quality circuit diagrams in LaTeX, using the circuitikz package. Take a look at the sample below:

The LaTeX code used to generate this diagram can be found at circuit-ups on GitHub. A simple example of constructing a circuit diagram is

  \begin{figure}
      \begin{circuitikz}

          \draw (0,0)
          to[V,v=$U_q$] (0,2) % The voltage source
          to[short] (2,2)
          to[R=$R_1$] (2,0) % The resistor
          to[short] (0,0);

      \end{circuitikz}

      \caption{My first circuit.}

  \end{figure}

and it creates the following circuit diagram:

Image Source: tutorial

All that is required are the circuitikz and siunitx packages which can be easily downloaded using tlmgr (the Tex-Live manager). On Ubuntu this can be done by running:

    tlmgr install circuitikz
    tlmgr install siunitx

I learned about using circuitikz from the following this tutorial which also contains a link to the circuitikz manual which came in handy.

All of the components in the circuit diagram on top are provided by the circuitikz package with the exception of the 7805 IC which I created by using primitive pgf directives I learned from this second tutorial. This was surprisingly easier than it sounds. And the output is very elegant.

(Note: The LaTeX commands generate pdf which I converted to a png using imagemagick before inserting it in to this blog)

Type-setting Urdu in LaTeX

I gained a considerable amount of experience working with LaTeX while I was completing my PhD. The thought immediately struck me that LaTeX was perfect for type-setting Urdu text because of the high demands for formatting that is required.

Turns out computer enthusiasts were way ahead of me. With the advent of XeTeX which brought unicode support to LaTeX and the bidi Tex-package which allows for bi-directional text (essential since Urdu is written from right to left) it is now possible to create beautiful urdu documents (pdfs), both prose and poetry, using LaTeX.

I have created a tutorial that explains how to do this in Ubuntu.

For samples of this technique:

nasr (prose)

nazm (poetry)

My book on using LeJOS with EV3 is now out

A little over a year ago when my wife gave me the Lego Mindstorms EV3 as the most awesome of birthday presents I started tinkering with it in an attempt to use Linux to program it. This led me to discover LeJOS. The subsequent joy of discovery moved me to create this blog and post what I learned about it.

These posts brought me to the attention of Packt publishers who commissioned me to write a book on the subject. That book has now been completed and is available on Amazon (amongst other places): Lego Mindstorms EV3 Essentials

The book is in essence a pedagogically rich instruction manual on how to program the EV3 using both the on-brick visual programming language and using Java by installing LeJOS. The primary focus of the book is using LeJOS while programming on a Linux machine. So if you are a robotics enthusiast who is comfortable with Linux (and really one cannot call themselves an enthusiast of any computer related field if one doesn't learn Linux) then you will love the detailed information this book provides on how to program the EV3 on that great OS. The fact that the EV3 runs Linux itself should be reason enough.

To get an idea of how the book presents its material you need go no further than this blog itself, which is the inception of the book, and the detailed information Packt have provided on their website: Packt Publishers - Lego Mindstorms EV3 Essentials.

SSH in to an EV3 running leJOS using the USB cable

I have underestimated leJOS at every turn, never more so than the issue of connectivity. I got Wifi to work by editing files directly on the SD card. That it worked is a testament both to leJOS and my own stubbornness. It turns out, however, that there is a much easier way of gaining SSH access to an EV3 running leJOS.

Simply use the mini-USB cable that ships with the EV3, connect the EV3 to your Linux machine and wait for the USB Ethernet connection to be established automatically (tested on Ubuntu where a desktop notification appears to indicate this). Run ifconfig to confirm that the usb0 network interface is up and running.

The EV3 displays its IP address on the home screen: 10.0.1.1. Simply SSH in to this address using username root and a blank password. Could it be more painless?

Connecting to an EV3 running LeJOS using Bluetooth

I had known all along that the EV3 comes equipped with Bluetooth and that LeJOS allowed you to write programs that employed Bluetooth for communication, but the whole truth is way cooler.
I learned only recently that with Bluetooth activated, LeJOS running on the EV3 implements a Bluetooth PAN (Personal Area Network) by becoming a GN (Group ad-hoc Network) Controller.
This means the EV3 is acting as a server on a bluetooth-based network. You can connect your computer to this network using bluetooth. The computer acts as a PANU (a client of the PAN) and is able to ssh in to the EV3 and perform all of the actions we were performing using the WiFi dongle. The immediate advantage is that one does NOT need the WiFi dongle (saves money and you don't have the dongle protruding off of the side of the EV3).
Bluetooth on Linux (my experience is with Ubuntu 12.04 and 13.04) is still buggy but the setup for connecting to the EV3 was surprisingly painless once I figured it out (that part was harder).
The first step is to confirm that bluetooth is running on your computer by running the following command in a terminal:

hciconfig -a

This will spit out a bunch of information:

hci0: Type: BR/EDR  Bus: USB
 BD Address: 00:16:CF:FE:03:A0  ACL MTU: 1017:8  SCO MTU: 64:8
 UP RUNNING PSCAN ISCAN 
 RX bytes:301012 acl:2298 sco:0 events:460 errors:0
 TX bytes:94808 acl:966 sco:0 commands:67 errors:0
 Features: 0xff 0xff 0x8d 0xfe 0x9b 0xfd 0x00 0x80
 Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3 
 Link policy: RSWITCH HOLD SNIFF PARK 
 Link mode: SLAVE ACCEPT 
 Name: 'Abid-Laptop'
 Class: 0x6e010c
 Service Classes: Networking, Rendering, Capturing, Audio, Telephony
 Device Class: Computer, Laptop
 HCI Version: 2.0 (0x3)  Revision: 0x20da
 LMP Version: 2.0 (0x3)  Subversion: 0x4182
 Manufacturer: Broadcom Corporation (15)

Look for "Networking" in 'Service Classes' and "Computer" in 'Device Class'. In addition we need a useful 'Name'. To change it issue:

sudo hciconfig hci0 name '<device name>'

To connect to the EV3 (running LeJOS) turn it on and navigate to the "Bluetooth" menu. Make sure that it says "Visibility on". Now on your computer issue:

hcitool scan

The response should look like:

Scanning ...
 00:16:53:40:E9:7D EV3

We will need the Bluetooth address of the EV3 (the 12 hexadecimal number interspersed with colons) and the IP address of the EV3 on the PAN (displayed on the EV3 screen - 10.0.1.1 in my case). To connect to the EV3 as a PANU client one uses the pand utility (from the bluez-compat package):

sudo pand --connect 00:16:53:40:E9:7D

The PAN has been established but we need to start the bnep0 network interface manually on the computer to allow us to access the EV3 over the bluetooth PAN. Since the EV3 has IP address 10.0.1.1 we choose for our computer 10.0.1.2 (the first three numbers (sub-domain) need to match, the last one can be any number from 2 to 255):

sudo ifconfig bnep0 10.0.1.2

To confirm that the bnep0 network interface is active simply run:

ifconfig

and look for:

bnep0     Link encap:Ethernet  HWaddr 84:a6:c8:9b:ce:58  
          inet addr:10.0.1.2  Bcast:10.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::86a6:c8ff:fe9b:ce58/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:670 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:30820 (30.8 KB)  TX bytes:428 (428.0 B)

To confirm connectivity with the EV3 we ping it:

ping -c3 10.0.1.1

One can ssh in to the EV3 using IP Address 10.0.1.1, username "root" and blank password:

ssh root@10.0.1.1

It helps with automation if you set up an SSH private/public key-pair and store the public key in /home/root/.ssh/authorized_keys allowing password-less access. Finally to disconnect the computer from the EV3 PAN issue:

sudo pand -K

Hello World C program on the EV3

I have been programming the EV3 using LeJOS (basically Java) for a few weeks now, but the desire to use a truly low-level language such as C to control the robot has not gone away. I took a significant first step in that direction by acquiring a cross-compiler and using it to successfully write and compile a Hello World program that basically prints the infamous string on stdout. Since my computer runs Linux on the x86 architecture while the EV3 runs on an ARM 9 architecture, a cross-compiler is needed on my x86 to take source code (in C) and compile it for the ARM 9 architecture. After compilation the executable can be placed in the EV3 and executed at will. Having a cross-compiler allows me to develop on my PC (a much faster approach) and simply push binaries (executables) to the EV3 for execution. This site/blog explains how to acquire the cross-compiler while this associated section explains how to write and compile the Hello World program. The blog has more complex examples to do with controlling the sensors and motors but that will have to wait for another day.

Using make to build and deploy LeJOS programs on the EV3

As discussed earlier my aim is to automate the repetitive nature of the edit-build-deploy-run cycle for programming the EV3 using LeJOS. To that end I set up build.gradle to build the project from the command-line as explained here. Furthermore I set up wifi on the EV3 (using a usb dongle). The next step was to edit ~/.ssh/config and add an entry for the EV3:

  Host ev3

      User root
      HostName 192.168.1.16
      IdentityFile ~/.ssh/abid_ev3

Note: Since the wifi setup on the EV3 is DHCP (not static) the IP address is liable to change, in which event one has to modify ~/.ssh/config accordingly.

The next step is to create a Makefile in your project root. Here is a link to the one I'm using for my "Rover" project. Note the extensive use of PHONY and EMPTY targets to ensure that build and scp tasks are only performed when needed. make once correctly set up takes care of the dependencies. I only need to issue make run from the terminal and the source is compiled and the jar file copied to the EV3 only if needed. Sweet and simple, just the way I like it.

You can take a look at the complete Rover source code on github. There you will also find a number of other repos for my various LeJOS EV3 projects.

Building LeJOS programs for EV3 using gradle

With the wifi connection (including ssh) to the EV3 running and IntelliJ set up to write programs, development for the EV3 was coming along nicely. There was still an issue, however, with the time it took to make a change in the code, compile it, copy it to the EV3 (using scp), run it on the EV3 and then repeat. Since repetition is the bane of all programmers I started looking in to ways to automate these tasks. Of course the ubiquitous make tool came instantly to mind. I would just write a make script (Makefile) to automate these tasks. But how to automate the build task that creates the jar file from the source code, and in such a way that make could do it.

The solution is gradle. Gradle is a project automation tool with excellent support for Java (Google now recommends using gradle and Android Studio - a port of IntelliJ - for Android development). Though gradle has somewhat of a learning curve, it is well worth it, since only after using gradle have I finally begun to understand how a java project comes together. Furthermore with gradle one can build projects from the command line by simply issuing the command gradle build.

Gradle installation is straightforward and is explained here. Once installed the next step is to create a build.gradle file in the root folder of your LeJOS project (your source code). The build.gradle for one of my earlier projects looks like this:

apply plugin: 'java'

def main_class = "Rover"          // We define the main class of the project

// We set the Version of the Java SDK to the one on the system: 1.6
sourceCompatibility = 1.6

sourceSets {

    main {

        // We specify the source directories for the project. The first one is created by us while the latter two are part of the LeJOS source and provide
        // essential classes for controlling the EV3.
        java {
            srcDirs = ['src', '/home/abid/applications/lejos/ev3/DBusJava/src', '/home/abid/applications/lejos/ev3/ev3classes/src']
        }

        // We specify the jar library required to compile the project. This provides the com.sun.jna package
        dependencies {
            compile files('/home/abid/applications/lejos/ev3/ev3classes/lib/jna-3.2.7.jar')
        }

        // We specify manifest attributes (key-value pairs that must appear in the META-INF/MANIFEST>MF file inside the output Jar for the EV3 to run it correctly.
        // Additionally we supply the name of the Main-Class which is the starting point of the program execution.
        jar {
            manifest {
                attributes("Main-Class": main_class, "Class-Path": "/home/root/lejos/lib/ev3classes.jar /home/root/lejos/libjna/usr/share/java/jna.jar")
            }
        }
    }
}

This is pretty much a template build.gradle file for any LeJOS EV3 project. The one-time alterations needed are to change the latter two srcDirs to point to the locations in the local LeJOS git repo as well as the jna jar dependency in the same.

The only thing one needs to alter (from project to project) is the name of the main class defined in the variable main_class near the beginning. And you are good to go. Execute gradle build and the project will compile and place the results in the build/ sub-folder.

Next time I will show you how to incorporate all of this in to one epic Makefile complete with dependencies.