mikejsavage.co.uk / blog

01 May 2017 / Pwn3d

lol I got a computer virus.

I noticed a WmiDrvSSE.exe burning 75% of my CPU, and promptly killed it. It immediately came back and went back to thrashing, so I looked a little harder. Everything says it's in C:\Windows\debug\WmiDrvSSE.exe, right click and get properties, not signed by Microsoft. Ok let's go look, it's not there but there is a PASSWD.log (apparently this is a normal Windows file?). Turn on the show system files option and there's a bunch of fucking DLLs like curl and iconv and winpthreads, so I rename the exe and kill it again, which stops it coming back.

Check my process list for anything else, winl0gon.exe, kill this shit, it immediately respawns itself, rename that file too (same folder) and kill it again and it stays dead. There's also an RegisterService.exe, so I check my services and sure enough there's a Windows FirewalI entry pointing at winl0gon.

I immediately assumed that a virus burning my CPU (cleverly it left one core idle so most people wouldn't notice it) would be running crypto ops, but fortunately virus total seems to think it was just a bitcoin miner. Checked the file creation date and they only got to mine for like 20 minutes.

The big question is of course, how did it get in? I was lucky enough to catch it within half an hour so I could remember what I was doing at it's file creation time: I was browsing the web like normal.

This is one of the things security conscious people get wrong a lot. Package signing is useless. Reproducible builds are useless. I have no real reason to be worried about my government attacking me through insane channels. Nobody is going to bother, when my web browser doubles up as an unauthenticated shell server.

I asked in the firefox IRC channel if there were any known exploits in 51.0.1 and was immediately chastised for being a few versions out of date. Lol. So my choice is malware, or constantly broken UI and extensions. (UPDATE: Firefox 52 drops support for ALSA on Linux too)

Does anyone know how to run Firefox in a sandbox? Like a real sandbox, not the useless tab sandbox Firefox already has built in, I want UAC dialogs every time it tries to read or write anything outside its installation directory, every time it tries to create a file. I could run whatever the fuck version of Firefox I want to and not have to worry about this.

For Googler's sake, the full list of files was:

C:\Windows\debug\libcurl-4.dll
C:\Windows\debug\libiconv-2.dll
C:\Windows\debug\libidn-11.dll
C:\Windows\debug\libintl-8.dll
C:\Windows\debug\libwinpthread-1.dll
C:\Windows\debug\zlib1.dll
C:\Windows\debug\RegisterService.exe
C:\Windows\debug\winl0gon.exe
C:\Windows\debug\WmiDrvSSE.exe

and all of them are system files so you have to go into folder options and turn those on to be able to see them.

If you have any comments email me and I will post the good ones.

22 Apr 2017 / Progress: libinput

UPDATE: rm /usr/share/X11/xorg.conf.d/40-libinput.conf and everything works again. Also probably don't read the rest of this because it sucks.

Some idiots have rewritten the X input layer and called it "libinput".

Unsurprisingly this breaks all of your existing configs and tools (xset and the old synaptics configs don't work) with no benefit.

But that's not all: you can no longer change your mouse sensitivity!

You're supposed to be able to install a separate program, xinput, which prints out this:

Virtual core pointer                          id=2    [master pointer  (3)]
  - Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
  - Laview Technology Xornet II               id=9    [slave  pointer  (2)]
  - Laview Technology Xornet II               id=10   [slave  pointer  (2)]
  - Wacom Intuos PT S Pen stylus              id=11   [slave  pointer  (2)]
  - Wacom Intuos PT S Finger touch            id=12   [slave  pointer  (2)]
  - Wacom Intuos PT S Pad pad                 id=13   [slave  pointer  (2)]
  - Wacom Intuos PT S Pen eraser              id=18   [slave  pointer  (2)]
Virtual core keyboard                         id=3    [master keyboard (2)]
  - Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
  - Power Button                              id=6    [slave  keyboard (3)]
  - Power Button                              id=7    [slave  keyboard (3)]
  - Sleep Button                              id=8    [slave  keyboard (3)]
  - USB Keyboard                              id=14   [slave  keyboard (3)]
  - USB Keyboard                              id=15   [slave  keyboard (3)]
  - Eee PC WMI hotkeys                        id=16   [slave  keyboard (3)]
  - Laview Technology Xornet II               id=17   [slave  keyboard (3)]

and then you're supposed to run xinput --list-props XX where XX is all the ids, and it prints a huge pile of shit:

Device 'Laview Technology Xornet II':
        Device Enabled (152):   1
        Coordinate Transformation Matrix (154): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        libinput Accel Speed (286):     0.000000
        libinput Accel Speed Default (287):     0.000000
        libinput Accel Profiles Available (288):        1, 1
        libinput Accel Profile Enabled (289):   1, 0
        libinput Accel Profile Enabled Default (290):   1, 0
        libinput Natural Scrolling Enabled (291):       0
        libinput Natural Scrolling Enabled Default (292):       0
        libinput Send Events Modes Available (271):     1, 0
        libinput Send Events Mode Enabled (272):        0, 0
        libinput Send Events Mode Enabled Default (273):        0, 0
        libinput Left Handed Enabled (293):     0
        libinput Left Handed Enabled Default (294):     0
        libinput Scroll Methods Available (295):        0, 0, 1
        libinput Scroll Method Enabled (296):   0, 0, 0
        libinput Scroll Method Enabled Default (297):   0, 0, 0
        libinput Button Scrolling Button (298): 2
        libinput Button Scrolling Button Default (299): 2
        libinput Middle Emulation Enabled (300):        0
        libinput Middle Emulation Enabled Default (301):        0
        Device Node (274):      "/dev/input/event23"
        Device Product ID (275):        9494, 43
        libinput Drag Lock Buttons (302):       <no items>
        libinput Horizontal Scroll Enabled (303):       1

Note that none of those are mouse sensitivity. Google says it's supposed to be called Device Accel Constant Deceleration (wtf), but that's not there.

I'm especially annoyed with the distro maintainers for this. When there's a breaking change coming, it's everyone's responsibility to push back against it. When you have tens of thousands of users, even small breaking changes add up to multiple man years of effort. You have to ask, "is this update worth several millions of dollars of people's time?"

When the answer is no, as a developer you scrap it and do something better. As a middle man (distro maintainers etc) you tell them to go away and do better. As a user, you are fucked because nobody listens to you.

Jesus christ.

If you have any comments email me and I will post the good ones.

11 Apr 2017 / bug489729

bug489729 was an awesome Firefox extension that disabled the shit where dragging a tab (which happens all the time by accident) off the tab bar causes it to open in a new window (which takes like a full second on a 6700K and makes all my windows resize)

(and of course you can't turn it off without a fucking extension)

I'm rehosting it here, mostly for my own convenience: bug.xpi

If you have any comments email me and I will post the good ones.

01 Apr 2017 / C++ tricks: better casting

C style casts are not awesome. Their primary use is to shut up conversion warnings when you assign a float to an int etc. This is actually harmful and can mask actual errors down the line when you change the float to something else and it starts dropping values in the middle of your computation.

Some other nitpicks are that they are hard to grep for and can be hard to parse.

In typical C++ fashion, static_cast and friends solve the nitpicks but do absolutely nothing about the real problem. Fortunately, C++ gives us the machinery to solve the problem ourselves. This first one is copied from Charles Bloom:

template< typename To, typename From >
inline To checked_cast( const From & from ) {
        To result = To( from );
        ASSERT( From( result ) == from );
        return result;
}

If you're ever unsure about a cast, use checked_cast and it will assert if the cast starts eating values. Even if you are sure, use checked_cast anyway for peace of mind. It lets you change code freely without having to worry about introducing tricky bugs.

Another solution is to specify the type you're casting from as well as the type you're casting to. The code for this is a bit trickier:

template< typename S, typename T >
struct SameType {
        enum { value = false };
};
template< typename T >
struct SameType< T, T > {
        enum { value = true };
};

#define SAME_TYPE( S, T ) SameType< S, T >::value

template< typename From, typename To, typename Inferred >
To strict_cast( const Inferred & from ) {
        STATIC_ASSERT( SAME_TYPE( From, Inferred ) );
        return To( from );
}

You pass the first two template arguments and leave the last one to template deduction, like int a = strict_cast< float, int >( 1 ); (which explodes). I've not actually encountered a situation where this is useful yet, but it was a fun exercise.

Maybe it's good for casting pointers?

If you have any comments email me and I will post the good ones.

25 Mar 2017 / Least effort unit tests

I wanted a C++ unit testing library that isn't gigantic and impossible to understand, doesn't blow up compile times, doesn't need much boilerplate, doesn't put the testing code miles from the code being tested, and doesn't have its own silly build requirements that make it a huge pain in the ass to use. Unfortunately all the C++ testing libraries are either gigantic awful monoliths (e.g. googletest), or tiny C libraries that are a little too inconvenient to actually use (e.g. minunit).

Ideally it wouldn't give you awful compiler errors when you get it wrong but that's probably impossible.

(caveat: I didn't actually look very hard at existing work because it would take more time than just writing my own)

Behold:

#pragma once

#if defined( UNITTESTS )

#include <stdio.h>

#define CONCAT_HELPER( a, b ) a##b
#define CONCAT( a, b ) CONCAT_HELPER( a, b )
#define COUNTER_NAME( x ) CONCAT( x, __COUNTER__ )

#define AT_STARTUP( code ) \
        namespace COUNTER_NAME( StartupCode ) { \
                static struct AtStartup { \
                        AtStartup() { code; } \
                } AtStartupInstance; \
        }

#define UNITTEST( name, body ) \
        namespace { \
                AT_STARTUP( \
                        int passed = 0; \
                        int failed = 0; \
                        puts( name ); \
                        body; \
                        printf( "%d passed, %d failed\n\n", passed, failed ); \
                ) \
        }

#define TEST( p ) \
        if( !( p ) ) { \
                failed++; \
                puts( "    FAIL: " #p ); \
        } \
        else { \
                passed++; \
        }

#define private public
#define protected public

#else

#define UNITTEST( name, body )
#define TEST( p )

#endif

It uses the nifty cinit constructor trick to run your tests before main, and you can dump UNITTESTs anywhere you like (at global scope). Example usage:

#include <stdio.h>
#define UNITTESTS // -DUNITTESTS, etc
#include "ggunit.h"

int main() {
        printf( "main\n" );
        return 0;
}

UNITTEST( "testing some easy stuff", {
        TEST( 1 == 1 );
        TEST( 1 == 2 );
} );

UNITTEST( "testing some more easy stuff", {
        for( size_t i = 0; i <= 10; i++ ) {
                TEST( i < 10 );
        }
} );

which prints:

testing some easy stuff
    FAIL: 1 == 2
1 passed, 1 failed

testing some more easy stuff
    FAIL: i < 10
10 passed, 1 failed

hello

It would be great if you could put UNITTESTs in the middle of classes etc to test private functionality, but you can't and the simplest workaround is #define private/protected public. Super cheesy but it works. It's not perfect but it works. It's ugly. It's not a Theoretically Awesome Injection Framework Blah Blah. It works and is two lines, you can't beat it.

If you have any comments email me and I will post the good ones.

23 Mar 2017 / Caches are fast, hashes are fast

Or, how to make a C/C++ build system in 2017

Here's a problem I've been having at work a lot lately:

Obviously the dream solution here would be to have good compilers(*) and/or a good language, but neither of those are going to happen any time soon.

*: as an aside that would solve one of the big pain points with C++. Everybody goes to these massive efforts splitting up the build so they can build incrementally which introduces all of its own headaches and tracking dependencies and making sure everything is up to date and etc and it's just awful. If compilers were fast we could just build everything at once every time and not have to deal with it.

Anyway since they aren't going to happen, the best we can do is throw computing power at the problem. Most of them build some kind of dependency graph, e.g. bin.exe depends on mod1.obj which depends on mod1.cpp, then looks at what has been modified and recompiles everything that depends on it. Specifically they look at the last modified times and if a file's dependencies are newer than it then you need to rebuild.

Maybe that was a good idea decades ago, but these days everything is in cache all of the time and CPUs are ungodly fast, so why not take advantage of that, and just actually check if a file's contents are different? I ran some experiments with this at work. We have 55MB of code (including all the 3rd party stuff we keep in the repo - btw the big offenders are like qt and the FBX SDK, it's not our codebase with the insane bloat), and catting it all takes 50ms. We have hashes that break 10GB/s (e.g. xxhash), which will only add like 5 ms on top of that. (and probably much closer to 0 if you parallelise it)

So 55ms. I'm pretty sure we don't have a single file in our codebase that builds in 55ms.

From this point it's pretty clear what to do: for each file you hash all of its inputs and munge them together, and if the final hash is different from last time you rebuild. Don't cache any of the intermediate results just do all the work every time, wasting 55ms per build is much less bad than getting it wrong and wasting minutes of my time. Btw inputs should also include things like command line flags, like how ninja does it.

The only slightly hard part is making sure hashes don't get out of sync with reality. Luckily I'm a genius and solved that too: you just put the hash in the object file/binary. With ELF files you can just add a new section called .build_input_hash or something and dump it in there, presumably you can do the same on Windows too (maybe IMAGE_SCN_LNK_INFO? I spent a few minutes googling and couldn't find an immediate answer).

For codegen stages you would either just run them all the time or do the timestamp trick I guess, since we are ignoring their timestamps and hopefully your codegen is not slow enough for it to matter very much.

Anyone want to work on this? I sure don't because my god it's boring, but I wish someone else would.

UPDATE: I've been told you can work around my specific example with git branch and git rebase --onto (of course), but this would still be nice to have.

If you have any comments email me and I will post the good ones.

02 Mar 2017 / C++ tricks: ZERO

After writing memset( &x, 0, sizeof( x ) ); for the millionth time, you might start to get lazy and decide it's a good idea to #define ZERO( p ) memset( p, 0, sizeof( *p ) );. This turns out to be very easy to misuse:

int x;
ZERO( &x ); // cool
int y[ 8 ];
ZERO( &y ); // cool
ZERO( y ); // y[ 0 ] = 0, no warnings
int * z = y;
ZERO( z ); // y[ 0 ] = 0
ZERO( &z ); // z = NULL

You can try things like making ZERO take a pointer instead, but you still always end up with cases where the compiler won't tell you that you screwed up. The problem is that there's no way for ZERO to do the right thing to a pointer because it can't know how big the object being pointed at is. The simplest solution is to simply not allow that:

template< typename T > struct IsAPointer { enum { value = false }; };
template< typename T > struct IsAPointer< T * > { enum { value = true }; };

template< typename T >
void zero( T & x ) {
        static_assert( !IsAPointer< T >::value );
        memset( &x, 0, sizeof( x ) );
}

and as a bonus, we can use the same trick from last time to make it work on fixed-size arrays too:

template< typename T, size_t N >
void zero( T x[ N ] ) {
        memset( x, 0, sizeof( T ) * N );
}

Neat! (maybe)

If you have any comments email me and I will post the good ones.

02 Mar 2017 / C++ tricks: safe ARRAY_COUNT

Lots of C/C++ codebases have a macro for finding the number of elements in a fixed size array. It's usually defined as #define ARRAY_COUNT( a ) ( sizeof( a ) / sizeof( ( a )[ 0 ] ) ), which is great:

int asdf[ 4 ]; // ARRAY_COUNT( asdf ) == 4

until someone comes along and decides that asdf needs to be dynamically sized and changes it to be a pointer instead:

int * asdf; // ARRAY_COUNT( asdf ) == sizeof( int * ) / sizeof( int ) != 4

Now every piece of code that uses ARRAY_COUNT( asdf ) is broken, which is annoying by itself, but that still looks totally fine to the compiler and it's not even going to warn you about it.

The fix is some appalling looking C++:

template< typename T, size_t N >
char ( &ArrayCountObj( const T ( & )[ N ] ) )[ N ];
#define ARRAY_COUNT( arr ) ( sizeof( ArrayCountObj( arr ) ) )

which correctly explodes when you pass it a pointer:

main.cc: In function "int main()":
main.cc:5:57: error: no matching function for call to "ArrayCountObj(int*&)"
 #define ARRAY_COUNT( arr ) ( sizeof( ArrayCountObj( arr ) ) )
                                                         ^
main.cc:9:9: note: in expansion of macro "ARRAY_COUNT"
  return ARRAY_COUNT( a );
         ^~~~~~~~~~~
main.cc:4:9: note: candidate: template<class T, long unsigned int N> char (& ArrayCountObj(const T (&)[N]))[N]
 char ( &ArrayCountObj( const T ( & )[ N ] ) )[ N ];
         ^~~~~~~~~~~~~
main.cc:4:9: note:   template argument deduction/substitution failed:
main.cc:5:57: note:   mismatched types "const T [N]" and "int*"
 #define ARRAY_COUNT( arr ) ( sizeof( ArrayCountObj( arr ) ) )
                                                         ^
main.cc:9:9: note: in expansion of macro "ARRAY_COUNT"
  return ARRAY_COUNT( a );
         ^~~~~~~~~~~

If you have any comments email me and I will post the good ones.

31 Jan 2017 / Dumping a git repository to an encrypted zip file

I want to be able to access my dotfiles repository from anywhere without actually giving people public access. I don't want to fuck about making a new user account with restricted shell/setting up a massive web server/etc.

The simplest solution I can think of is making a post-receive hook that dumps the repository to an encrypted zip and copying that to the (static) web root, which is done like this:

#! /bin/sh

OUT=/path/to/webroot/dotfiles.7z

rm "$OUT"
git archive master | 7z a -sidotfiles.tar -ppassword -mhe=on "$OUT"

Ok so it's a 7z not a zip, but some zip implementations (like Windows Explorer) only support shitty encryption so you were going to have to install 7zip anyway.

If you have any comments email me and I will post the good ones.

25 Jan 2017 / Windows post-install for developers

This is just a checklist for myself covering what to do with a fresh Windows installation. It covers disabling all the annoying crap Windows comes with by default, updating manually because Windows Update is broken in Windows 7 SP1, and a list of handy programs.

  1. Install drivers and reboot.
  2. Go to services.msc, stop and disable: Superfetch, Windows Defender, Windows Firewall, Windows Search.
  3. Go to Control Panel and view by small icons. Go to Administrative Tools, Computer Management, Local Users and Groups, Users, right click Administrator and enable it. Log in as Administrator. This is now your user account, and you can delete your old one.
  4. Right click the start menu, click properties, use small icons, never combine taskbar buttons, unlock, drag to left, lock. Under the start menu tab, click customise, then disable Devices and Printers, Games, Help, Highlight newly installed programs, Music, Pictures, and Use large icons.
  5. Click the start menu, right click Computer, Advanced system settings (in the sidebar), Startup and Recovery settings, disable Automatically restart. Close that window and go to Performance settings. Uncheck lots of crap.
  6. Right click the desktop, enable Windows Classic theme.
  7. Go to Control Panel, then Action Center and disable UAC (in the sidebar), then go to Change Action Center settings (also in the sidebar) and disable problem reporting and all the messages (except for maybe Windows Update). Go to AutoPlay and disable that too. Go to Mouse and disable enhanced pointer precision. Go to Sound and select the No Sounds sound scheme. Go to Power Options and disable monitor/PC sleeping.
  8. In the start menu, search for folder options. Go to view and show hidden files and known extensions.
  9. In Computer, right click your C drive, go to Security, Advanced, Change Permissions, click your name, Edit, check full control, click OK, check "Replace all child permissions...", click OK.
  10. Install Firefox.
  11. Install MSVC Community 2013. Uncheck all the optional features. This download is huge so start it first!
  12. Install the Windows 7 convenience rollup update and its dependencies.
  13. Download Autoruns, Process Explorer, and Process Monitor.
  14. Install Color Cop, AutoHotkey, 7-Zip, Everything.
  15. Install Git for Windows, Notepad++, and Vim.
  16. Install Renderdoc, Apitrace, Intel GPA, and the DirectX SDK.
  17. Install GIMP, Inkscape, Blender, Wings3D, and Meshlab.
  18. Download the Cygwin installer. Install tmux, openssh, lua and vim.
  19. In a cygwin shell, run ssh-host-config, and follow the prompts. chown cyg_server: /var/empty; chmod 700 /var/empty; net start sshd.

If you have any comments email me and I will post the good ones.

29 Dec 2016 / Billions

When people are trying to sell candidates on their company, they like to throw around statistics like "we do billions of Xs per year", or if they want to pull out the really big guns, "we do one billion Xs per day".

I (and presumably everyone else) hear these statistics and think "wow a billion is a big number that's impressive" and don't think too much more about it.

Until now! Let's figure out just how impressive these numbers are. Assuming 365 * 24 * 60 * 60 seconds in a year, one billion Xs per year is 32 Xs per second, which is actually not impressive at all. If we try again with one billion per day (86400 seconds in a day) we get 11.5k per second, which is back into impressive territory... until you think about it.

My crappy game engine on my laptop with integrated graphics does 1M verts per second without the fans spinning up. AAA games routinely do 100x that - 4 orders of magnitude higher than 1b/day.

I should sell my engine by telling people it can do QUADRILLIONS of verts per year!!!!!!!!

If you have any comments email me and I will post the good ones.

23 Apr 2016 / Auto-mounting removable drives

Put this in /etc/udev/rules.d/10-automount.rules:

KERNEL!="sd[c-z][1-9]", GOTO="media_by_label_auto_mount_end"

# Global mount options
ACTION=="add", ENV{mount_options}="relatime,users,sync"

# Filesystem specific options
ACTION=="add", PROGRAM=="/lib/initcpio/udev/vol_id -t %N", RESULT=="vfat|ntfs", ENV{mount_options}="$env{mount_options},utf8,gid=100,umask=002"
ACTION=="add", PROGRAM=="/lib/initcpio/udev/vol_id --label %N", ENV{dir_name}="%c"
ACTION=="add", PROGRAM!="/lib/initcpio/udev/vol_id --label %N", ENV{dir_name}="usbhd-%k"
ACTION=="add", RUN+="/bin/mkdir -p /mnt/%E{dir_name}", RUN+="/bin/mount -o $env{mount_options} /dev/%k /mnt/%E{dir_name}"
ACTION=="remove", ENV{dir_name}=="?*", RUN+="/bin/umount -l /mnt/%E{dir_name}", RUN+="/bin/rmdir /mnt/%E{dir_name}"
LABEL="media_by_label_auto_mount_end"

You need to change the first line (specifically the [c-z] bit) if you have more (or less) than two non-removable drives. I don't know exactly how it works, but it does the job. I copied it from the arch wiki years ago and I'm putting it here for my own reference.

If you have any comments email me and I will post the good ones.

27 Dec 2015 / Moving to OpenBSD

My last VPS provider got bought out, so I figured now would be a good time to buy a more respectable (i.e. not found on lowendstock) server, which also gives me the opportunity to experiment with OpenBSD.

As expected, almost everything has gone smoothly. However, there have been a couple of pain points, which I'll document here for future me and any lucky Googlers.

tinc

If you copy and paste a working config from a Linux server, the clients can all ping the server, but you get an error like "No route to host" when you try it the other way around. This turned out to be a pf one-liner:

pass out on tun0 to 10.0.0.0/8

(where tun0 is my VPN's interface and 10.0.0.0/8 is my VPN's subnet)

luarocks

There isn't a luarocks port yet, and building can be slightly annoying. It goes like this:

./configure --sysconfdir=/etc/luarocks \
    --lua-version=5.1 \
    --lua-suffix=51 \
    --with-lua-include=/usr/local/include/lua-5.1
make build
make install

The configure lines for 5.2 and 5.3 look like you would expect. The install step creates a symlink from luarocks to luarocks-5.x so if you are lazy like me you should correct that to point at your favourite version.

lua-ev

The latest stable release doesn't work with Lua 5.3, and for some reason the lua-ev rock doesn't look in /usr/local/include. We can fix the former by installing the scm version, and the latter with CFLAGS:

luarocks install \
    https://luarocks.org/manifests/brimworks/lua-ev-scm-1.rockspec \
    CFLAGS=-I/usr/local/include

If you have any comments email me and I will post the good ones.