Mounting USB devices in WSL-2

My goal was to deduplicate all the images and videos I have in my large backup USB hard drive. Obviously this was a task for Linux but the only Linux I have ready access to these days is WSL-2 running under Windows 11.

By default USB devices connected on the Windows side are not accessible from WSL-2. The solution is laid out in this how-to from Microsoft.

Setup

Prequisite: WSL needs to have kernel which is USBIP capable. This can be achieved by updating WSL in Powershell: wsl --update.

Windows side

Step 1: Install the (open-source) usbipd-win utility (msi) from the project's releases page. Verify the installation by launching Windows Powershell and running usbipd list.

WSL side

We now need to compile the Kernel to enable CONFIG_USB_STORAGE.

Step 1: Figure out the current kernel version. In Powershell run: wsl --version. In our example the kernel version is 5.15.153.1-2.

Step 2: Clone the WSL kernel source code for the version installed: git clone --depth 1 --branch linux-msft-wsl-5.15.153.1 https://github.com/microsoft/WSL2-Linux-Kernel.git

Step 3: Install build tools: sudo apt-get install git bc build-essential flex bison libssl-dev libelf-dev dwarves

Step 4: Install usbip tools on the WSL side: sudo apt-get install usbutils hwdata

Step 5: 

cd WSL2-Linux-Kernel

cp Microsoft/config-wsl .config

scripts/config --enable USB_STORAGE

make oldconfig

The last command will prompt for a bunch of information. Answer y to everything except "verbose debug" (and possibly Rio karma music player).

make -j$(nproc)

cp arch/x86/boot/bzImage /c/Users/<windows-user>/

Step 6: Modify wsl config to use this compiled kernel by modifying the /c/Users/<windows-user>/.wslconfig file and pointing the kernel property to C:\\Users\\<windows-user>\\bzImage

Step 7: Restart WSL (in Powershell run: wsl --shutdown and then launch WSL again)

Sources:

• https://renenyffenegger.ch/notes/Windows/Subsystem-for-Linux/index#wsl-compile-kernel
• https://renenyffenegger.ch/notes/Windows/Subsystem-for-Linux/index

Mounting USB device in WSL

Step 1: In Powershell run usbipd list and take a look at the devices that are already attached. We will rerun this command and look at the change to discover our device.

Now attach your USB device and eject it (while keeping it plugged in). Attachment will fail if Windows is accessing the USB device.

Step 2: In Powershell run usbipd list and determine the busid (BUS ID) of the attached device. It is of the form d-d (where d is an integer). In our example the busid will be 2-6.

We will now bind the USB devise to WSL (make sure that the WSL VM is up, opening a WSL terminal should suffice). 

Step 3: In an Admin Powershell: usbipd bind --busid 2-6. To verify run usbipd list and check that the STATE for your device now says Shared.

Step 4: In (a normal) Powershell: usbipd attach --wsl --busid 2-6. Verify by running usbipd list and checking that the STATE for your device now says Attached.

Step 5: In WSL run lsusb to see the USB device showing up. Then run dmesg and look at the last bit of output. We are looking for usb, scsi, and sd output, in particular any mention of a disk partition e.g. sdd1.

Step 6: Mount the partition: mount /dev/sdd1 /mnt/usb

Unmounting USB device in WSL

Step 1: Unmount partition in WSL: umount /mnt/usb

Step 2: Unattach device in Powershell: usbipd unattach --busid 2-6

Step 3: Unbind device in Powershell: usbipd unbind --basuid 2-6

Installing, configuring and using tin to read Usenet articles in Linux (WSL-2)

 I recently learned about Gemini and how its development is being tracked on Usenet. This got me interested in accessing Usenet.

Registering with Usenet Provider

The first step is to register with a Usenet provider. Since I am primarily interested in the comp.infosystems.gemini group I can use a free provider which only gives text access. I chose to use https://www.eternal-september.org/. Registration was extremely straight forward and I received credentials via email.

Installing tin (Usenet reader)

I looked around for a Linux based Usenet reader and came across tin. I am currently using a Windows laptop with WSL-2 installed running Ubuntu 20.04. I installed tin using:

sudo apt-get install tin

Authenticating with tin

I created a ~/.newsauth file readable only by myself (chmod go-r ~/.newsauth) and placed my credentials inside it in the recommended format:

news.eternal-september.org <password> <username>

All of this information came from the registration email I got from eternal-septemeber.

Configuring tin

To use neovim as the editor of choice when replying to articles I set export EDITOR='nvim' in my ~/.zshrc. I also set TIN_HOMEDIR="$HOME/.tin" to consolidate the various configuration files.

To correctly identify myself when posting articles (replies) I modified ~/.tin/tinrc and set mail_address="Abid H. Mujtaba" <abid.mujtaba@protonmail.com>

Running tin

Launch via: tin -r -g news.eternal-september.org -A comp.infosystems.gemini

-r tells tin to fetch from articles from the remote server (rather than local). -g is used to specify which remote server to use (eternal-september in my case), and -A tells tin to authenticate with the remote server (using the credentials in ~/.newsauth, prompts for creds if that file doesn't exist).

One can set NNTPSERVER="news.eternal-september.org" to avoid having to specify -g. Similarly one can (initially) place the line comp.infosystems.gemini: (note the necessary colon at the end) in ~/.tin/.newsrc (extra info will be appended after the colon automatically as you interact with the usenet group). With these changes one can simply issue tin -r -A.

Using tin

To get started, select a thread (of articles). To read the next article in the thread click n (p for the previous article). Use j and k to move down and up inside an article. Can also enter the article number (within the thread) to jump to it.

To post in response to an article use f (follow-up). This is different then replying (r) which sends an email to the author (using sendmail).

 

Compiling Latex with latexmk with tikz externalization and pre-compiled preamble (which includes tikz)

Compilation of LaTeX projects that contain tikz images can be sped up by an astronomical amount by properly separating the static and dynamic preamble and issuing the right commands.

A minimum-working-example that I wrote that demonstrates this in great detail can be found at: https://github.com/abid-mujtaba/mwe-latexmk-tikz-preamble

Ubuntu - Copy large number of files easily to Android Phone

Situation:

The MTP protocol for transferring files to and from an Android Phone is utter garbage, especially when transferring a large number of files (like your Music collection for the first time - on my Google Pixel). The connection inevitably fails in the middle of the transfer, which is slow to boot.

Solution:

USB tether your phone to your computer (read http://abidmujtaba.blogspot.com/2017/07/ubuntu-usb-tethering-with-google-pixel.html if you get stuck). You don't even need internet pass-through, just an IP Address (so you can use sudo dhclient usb0 instead of sudo dhcpcd usb0 if you are using the terminal).

Next follow http://abidmujtaba.blogspot.com/2017/07/ubuntu-ssh-in-to-android-phone-google.html to set up an ssh connection. Once you have that you are home free.

I use lftp over ssh to copy all the files I want. I even have a simple lftp bookmark set up:

pixel sftp://abid:xx@pixel

where my ssh config section is titled "pixel" and the password "xx" is just a red-herring since ssh is configured to use an RSA key for authentication. This bookmark allows me to access my phone's files using a simple lftp pixel.

Ubuntu - SSH in to Android Phone (Google Pixel)

The ability to SSH in to your (unrooted) Android Phone is a very powerful one to have, and it is surprisingly simple.

1. Install the "SimpleSSHD" app on your phone (form the Google Play Store).
2. Copy your PUBLIC ssh key (form your computer) to the "/sdcard/ssh" (default) folder on your phone and rename it authorized_keys.
3. Run the "SimpleSSHD" app and click start (uses port 2222 be default since it runs in user space). The app will tell you the IP address of the phone.
4. SSH in to your phone by issuing:

   ssh -o StrictHostKeyChecking=no -p 2222 -i <private key location> <phone ip address>

   You will need to issue StrictHostKeyChecking=no the first time only.
5. To avoid having to type all of this out every time you can add a section to the ssh config file (~/.ssh/config).

   Host pixel
   
       Hostname 192.168.42.129
       Port 2222
       IdentityFile ~/.ssh/pixel
   

   and simply run ssh pixel to connect.

Nore: This technique works with USB tethering as well. SimpleSSHD will show both the WiFi and the USB Tethering IP Addresses and you can use the latter for super-fast ssh.

Ubuntu - USB Tethering with the Google Pixel

Situation:

When I connected my Google Pixel to my Ubuntu (14.04) laptop with the USB cable and selected the option to USB tether on the phone the connection would fail. The Network Manager applet would show a greyed out option "Ethernet Network (Google Pixel)" with the status "disconnected" underneath it, also greyed out.

Interestingly running "ifconfig" revealed that a "usb0" section was present but no IP address had been assigned.

usb0      Link encap:Ethernet  HWaddr 32:a8:e6:6c:fe:9d  
          inet6 addr: fe80::30a8:e6ff:fe6c:fe9d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:560 (560.0 B)  TX bytes:1802 (1.8 KB)

This clue led to a long trawl across the internet that led to a surprisingly simple solution.

Solution:

Install the "dhcpcd" package.

sudo apt-get install dhcpcd

Now after connecting the Google Pixel simply run:

sudo dhcpcd usb0

And confirm by running:

ifconfig usb0
route -n
ping -c3 www.google.com

Compiling Latex with scons with tikz externalization and pre-compiled preamble (which includes tikz)

Edit: I have created another MWE that uses latexmk instead of scons (which I found to be using more resources and not recompiling correctly when citations were added): http://abidmujtaba.blogspot.com/2017/10/compiling-latex-with-latexmk-with-tikz.html

Compilation of LaTeX projects that contain tikz images can be sped up by an astronomical amount by properly separating the static and dynamic preamble and issuing the right commands.

A minimum-working-example that I wrote that demonstrates this in great detail can be found at: https://github.com/abid-mujtaba/mwe-scons-tikz-preamble