Roger Binns — Tue 08 December 2020

Python 2 has come to an end. I ported the last of my personal scripts to Python 3 a few months ago.

Perhaps the greatest feature of Python 2 was that after the first few releases, it stayed stable. Code ran and worked. New releases didn't break anything. It was predictable. And existing Python 2 code won't break for a long time.

The end of Python 2 has led to the end of that stability, which isn't a bad thing. Python 3 is now competing across a broader ecosystem of languages and environments trying to improve developer and runtime efficiency. Great!

I did see a quote that Python is generally the second best solution to any problem. That is a good summary, and shows why Python is so useful when you need to solve many different problems. It is also my review of Python 2.

So let's have some musings ...

Python has had poor timing. The first Python release (1994) was when unicode was being developed, so the second major Python version (2000) had to bolt on unicode support. But if it had waited a few more years, then things could have been simpler by going straight to utf8 (see also PEP 0538).

Every language has been adding async with Python 3 (2008) increasing support with each minor release. However like most other languages, functions ended up coloured. This will end up solved, almost certainly by having the runtime automagically doing the right thing.

Python 3 made a big mistake with the 2to3 tool. It works exactly as described. But it had the unfortunate effect of maintainers keeping their code in Python 2, and using that to make releases that supported both Python 2 and 3. The counter-example is javascript where tools provide the most recent syntax and transpiling to support older versions. Hopefully future Python migration tools will follow the same pattern so that code can be maintained in the most recent release, and transpiled to support older versions. This should also be the case for using the C API.

The CPython C API is quite nice for a C based object API. Even the internal objects use it. It followed the standard pattern of the time with an object (structure) pointer and methods taking it as a parameter. There are also macros for "optimised access". But this style makes changing underlying implementation details difficult, as alternate Python interpeter implementations have found out. If for example a handle based API was used instead, then it would have been slower due to an indirection, but allow easier changing of implementation details.

Another mistake was not namespacing the third party package repository PyPI. Others have made the same mistake. For example when SourceForge was a thing, they did not use namespacing so the urls were sf.net/projectname - which then led to issues over who legitimately owned projectname. Github added namespaces so the urls are github.com/user/projectname. (user can also be an organization.) This means the same projectname can exist many times over. That makes forking really easy, and is perhaps one of the most important software freedoms.

Using NPM as an example, this is the only package that can be named database. It hasn't been updated in 6 years. On PyPI this is apsw and hasn't been updated in 5 years. (I am the apsw author updating it about quarterly but not the publisher on PyPI for reasons.) Go does use namespacing. A single namespace prevents forks (under the same name) and also makes name squatting very easy. Hopefully Python will figure out a nice solution.

Category: misc – Tags: exit review, python

Roger Binns — Sat 14 September 2019

I'm a fan of podcasts and especially longer form history podcasts. I've found that "History of" podcasts that cover various empires and locations seem to be rather good. The History of Rome podcast is a very good example, with many others following that format and principles. The format allows the shows to adapt over time, include listener feedback, and do experiments which often work well.

If you can't get enough, then Hardcore History has many good episodes and stories.

And at the meta level, there is a History of *History of podcast* podcasts

Category: misc – Tags: recommendation

Roger Binns — Wed 10 April 2019

Category: misc – Tags: review

Roger Binns — Thu 28 February 2019

I've been wondering what best practise for handling defaults is. In software there are generally 3 values: zero, one, or many. As a consequence developers often pick a sensible number for "many", and allow configuration to change it.

Eventually defaults permeate the code, settings, user interfaces, product documentation, user forums, and search engine results. It spreads not from a single source of truth that tracks and propagates changes, but by being arbitrarily copied between systems.

As time passes, the default values need to change due to circumstances and experience. New features make existing values need refinement, while new interactions complicate matters.

The usual solution is to bump the major version and have humans, code, and documentation deal with changes. The effort of doing major version upgrades especially all the setting changes is what makes so many of us resistant to do major version upgrades.

Starting software after a version upgrade is always a pain. Sometimes you are pleasantly surprised that it just works, but usually the logs are full of complaints about settings, things that previously worked no longer working and general yak shaving.

Postfix has a compatibility level to help defer the effort after a major version upgrade, but you are still on the hook for the upgrade changes.

An anti-pattern is software that generates an initial config file for you. It does have a very short path between default settings and the generated config file, usually including comments and explanations in that file. This is fantastic to start with.

But it causes problems in time. The settings, comments and explanations become wrong. Looking at a config file that is a few years old is an exercise in archaeology and contradictions, requiring consulting the file, warning/error messages, logs, wikis, and other documentation.

So far the best I have is to prefer more 'automatic' settings, and keep the number of settings to a minimum.

Category: misc

Roger Binns — Wed 18 November 2015

A shocking time has come - I've given up Emacs, after using it for 20 years. When interviewing developers, one of the questions I ask is about their favourite editor. I don't care what the answer is, but I do very much care about why it is. An editor is a fundamental part of developer productivity, so I want to hear about the candidate caring about their own productivity and trying to improve it on an ongoing basis.

The irony is that I was using the same editor for decades. I did keep trying to find improvements, but never could. There are two sides to Emacs - one is as a competent & coherent editor, and the other is "living" in it. It has builtin web browsing, image viewing, email and news support, terminal emulators etc. I was never one of those.

Before Emacs I used vi. Its modal interface, small size, and availability on all systems make it a good tool. However it was text console only, and didn't do colour, menus, multiple files or other useful functionality. (It does now.) vi does have a learning curve - I estimate it takes about 4 years to be good with it, and 8 years to be an expert!

I had known about Emacs for a while, but it was text console only, and didn't do colour, or menus. Each attempt to use it left me frustrated with what amounts to another arbitrary set of keystrokes. (I've always been a cross platform person so I was also juggling keystrokes for other operating systems and applications.) A colleague (hi Jules) introduced several of us to XEmacs around 1995. It had a gui, and colour, and most importantly a menu system. It was no longer necessary to memorize a large set of new keystrokes, as the menus showed them. You could do everything without knowing any, and then pick up those you use often enough.

By the mid 2000s XEmacs was languishing, and Emacs was slowly catching up with the gui. More and more packages only worked with regular Emacs (there were small but growing incompatibilities). I eventually made the switch from XEmacs to regular Emacs.

There was an explosion in different file types I was editing: Python, C, Javascript, Java, Objective-C, HTML, HTML with Jinja Templates, JSON, matlab, CSS, build scripts, SQL, and many more I have forgotten. Emacs had support for most. Support means syntax highlighting, indenting, jumping around notable symbols etc. More packages were produced that did linting (looking for common errors), and various other useful productivity enhancements.

At the same time a new editor Sublime Text was introduced. It had fantastic new interaction (goto anything, projects, command palettes, multiple selections, distraction free) and a rich package system (written in Python - yay!) I kept trying it, but kept finding issues that affected me. Development also seemed to drastically slow, and since it was closed source there was no way for others to improve and update the core.

Meanwhile Emacs became more and more frustrating. The web (HTML, Javascript, CSS) is not a first class citizen. Not many packages were distributed with the core, and you had to copy crytic elisp code from various places, or use strange tools to try and get them installed and kept up to date. Then you had to do that on each machine. Heck the package repositories (eg MELPA) didn't even use SSL by default! My emacs configuration file kept getting longer and longer.

Ultimately tools these days are defined by their vibrant community, useful defaults, and easy to use extension mechanisms. Emacs has all those, especially in the past. But they are of a different era and different cadence.

I have switched to Atom. It had a rough initial exposure with performance problems, and the extremely dubious choice of being closed source. However both have been addressed. Just days before Atom 1.2 was released, I removed Emacs in favour of Atom 1.1. My configuration file is 10 lines long, and I get the same experience on every machine.

Category: misc – Tags: exit review

Roger Binns — Thu 30 July 2015

I am one of the many many people getting the completely useless response on trying to upgrade to Windows 10.

Closing setup and trying again doesn't work. Nor did changing my language settings, .Net repair tools, freeing up huge amounts of space, rebooting, examining log files, or reading tea leaves.

A lesson I learned many years ago is that developers should work in support [1] (eg 3 weeks a year). Normally their experience of support issues is after they have been filtered through many layers of other people, and they don't see the ones that have been resolved even if fairly frequent. Essentially developers do not experience the friction that their customers or support staff regularly encounter.

I've done the support work myself, as well as seeing other developers doing it. There are immediate fixes such as tweaks to tools, asking for information in a different order, or new ideas for how to address common issues (hopefully eliminating them). Then they will go back and fix unhelpful messages in the product. For example a message like "couldn't find file" will be changed to say which file, and possibly detect if it is because the file doesn't exist or the directory containing it. Or make the code create a default file and parent directories.

From that point on, the developers produce substantially better diagnostics. They work out what information they would need on answering a support call and make the diagnostics provide it all. But as time passes, the memories fade, and shortcuts are taken. That is why working in support should be done regularly.

The Microsoft developers responsible for the screenshot "something happened" are likely a lost cause though.

As a side rant, Linux distributions are also distributed as ISO files. They can be used as is on optical media, network booted, dumped as is onto USB flash drives, and work on BIOS and EFI systems (even Apple's non-standard EFI). The Windows ISO is considerably more painful, especially if your machine doesn't have an optical drive, like pretty much all of them these days.

Category: misc – Tags: rants

Roger Binns — Tue 19 May 2015

Fundamentals

I recently decided to get a multipurpose device. They can do all this:

• Large battery to recharge other devices over USB (eg your phone and tablet)

• Provide wifi access to a network in front of it

  A wifi network is provided behind the device with a name and password of your choosing. You connect one or more of your devices to that.

  In front of it you can have no network at all, a wifi network (unrelated to the one behind), or a wired ethernet network. You do have to configure access to the network, but only for this device. Your devices are behind it blissfully ignorant of the real network.

• Exports attached storage (eg USB stick, USB hard drive, sdcard) via both SMB (aka "Windows network file sharing" supported by virtually everything these days) and DLNA (a multimedia network protocol, supported by many although the Apple ecosystem prefers "iTunes")

  On Android and desktop systems, you'll find that Kodi works for both SMB and DLNA, as does Android ES File Explorer (SMB only).

• They are cheap ($40 - $60 depending on battery capacity)

• Can run completely off the battery so no additional power is needed. They will run for many hours. They will also run while being charged.

• Similar in size to a pack of cards

• They use popular standards - eg they charge using standard micro-USB, provide power for devices with standard USB port, use existing filesystems, standard protocols etc. There is no need to carry different cables or chargers, and any software speaking SMB or DLNA works.

After some agonizing over Amazon reviews, and reading the manuals, I ended up with two.

The left red one is a HT-TM05 TripMate Versatile Wireless N Travel Router (Amazon page) although the packaging and internal names say Tripmate Sith. The right white one is a RAVPower RP-WD02 Wireless Filehub / Portable Travel Router (Amazon page). They are sold by the same company, and the underlying products are substantially similar except for the hardware layout.

How they do all this turns out to be quite simple. The battery provides power, and there is a small Linux based computer attached. It is running a MIPS based processor (the manuals even tell you the exact manufacturer and model number), 32MB of RAM, and 8MB of builtin storage for their software. For some reason MIPS cores seem very popular in network access devices - if you have a box at home from the likes of Linksys, DLink, Netgear etc, it is almost certainly using MIPS.

$ cat /proc/version
Linux version 2.6.36 (gcc version 3.4.2) #8 Fri Jul 11 10:44:45 CST 2014
$ /usr/sbin/smbd --version
Version 3.0.24

$ cat /proc/version
Linux version 2.6.21 (gcc version 3.4.2) #5 Fri Nov 1 13:36:46 CST 2013
$ /usr/sbin/smbd --version
Version 3.0.24

3. You may copy and distribute the Program (or a work based on it, under
   Section 2) in object code or executable form under the terms of Sections 1
   and 2 above provided that you also do one of the following:

   a) Accompany it with the complete corresponding machine-readable source
      code, which must be distributed under the terms of Sections 1 and 2
      above on a medium customarily used for software interchange; or,

   b) Accompany it with a written offer, valid for at least three years, to
      give any third party, for a charge no more than your cost of physically
      performing source distribution, a complete machine-readable copy of the
      corresponding source code, to be distributed under the terms of Sections
      1 and 2 above on a medium customarily used for software interchange; or,

   c) Accompany it with the information you received as to the offer to
      distribute corresponding source code. (This alternative is allowed only
      for noncommercial distribution and only if you received the program in
      object code or executable form with such an offer, in accord with
      Subsection b above.)

- Redistributions in binary form must reproduce the above copyright
  notice, this list of conditions and the following disclaimer in
  the documentation and/or other materials provided with the
  distribution.

Category: misc – Tags: review

Roger Binns — Tue 16 September 2014

(I was recently asked about Linux top. Here is my response.)

Back in the second half of 1991, someone posted about a new Unix like operating system. It was an instant hit for those of us lucky enough to be connected to the Internet. At my university's several computer labs we had good access to a variety of systems, especially large numbers of Sun workstations, some Windows/286 PCs, and a big Pyramid system shared by students and faculty.

If you wanted something Unix like on your own PC there wasn't much choice. The front runner was Minix which was very expensive for students. It was also for the 8086 and didn't take advantage of more than 1MB of RAM or memory protection, plus had single threaded filesystem access making that slow too. There were a big set of patches for the 80386, but they weren't officially supported.

Linus started out with some assembly language hosted by his Minix system that explored the 80386 task switching instructions. It didn't take too much more before it could run processes and access the filesystem. It was just for fun. It was simple, effective and just what so many people were looking for with no barriers to participation. Releases were frequent as people scratched their own itches.

One thing the fledgling Linux didn't have is getting a list of running processes. (It also didn't have sound, mice, networking, gui, or a whole bunch of other normal stuff yet so that wasn't a big omission.) You had to press the Scroll Lock key and it would spew some stuff on the console showing the running processes. That wasn't a huge burden since it only supported 63 processes total. You can still see the descendent of the scroll lock diagnostics by using the SysRq key with t for tasks.

It was also very common to download the source for various parts of the system as they frequently changed to keep up to date with system calls and semantics. For example this included bootloaders, compilers, C libraries, text editors etc. You'd want to recompile everything on each Linux kernel release which took a while. And often you'd want to know if something got stuck, or if things really were proceeding.

Branko Lankester released a version of ps that was a quick hack. Instead of adding "proper" support for getting a list of processes, he added an ioctl to the keyboard driver (since all systems had one making it easy to access).

While ps is nice, it only gives you a snapshot of the system and no idea about what is happening over time. I was already familiar with top and wanted that to monitor the system. The existing top had no hope of working on Linux so I wrote one from scratch. It was pretty much the simplest way of wrapping ps so that it sorted the processes, and then output as many would fit on the screen. That is why the documentation always mentions Branko and his ps.

I did add some unique things over the top I had used. You could specify which fields were shown and in which order. There were less than 26 so I used the letters of the alphabet. It could be saved to a $HOME/.toprc so you didn't need to reconfigure each time you ran. The same configuration was preserved for many years by future authors.

I released my top in early 1992. I then had a 6 month work placement where I no longer had access to a machine I could run Linux on. Others then took over top, updating it as Linux matured, exported process information more sensibly, removed some inefficiencies I had coded, all leading up to what you have today. I would be shocked if any of the original code survives. I am still listed in the credits though.

top was developed on an 80386 system running at 20MHz with 4MB of RAM. It had a 100MB hard drive of which 80MB was used for DOS and 20MB was allocated for Linux. I used vi as the text editor, and had to figure out various library usage (eg dealing with the terminal) by reading library source code.

Category: geek – Tags: linux, history

Roger Binns — Mon 21 July 2014

The US penal system is despicable in many different ways. John Oliver covers some of it.

Category: misc – Tags: reality

Roger Binns — Sat 19 July 2014

When people find out that you pay for incoming cell phone calls in the US, it seems illogical and likely part of the various issues where the country is dysfunctional (eg politics, health care, rampant hypocrisy). However there are good reasons for this system and issues in your own country you may not be aware of.

Whenever a phone call is made there are termination rates - the amount paid to the operator on the receiving end [1]. In most countries the termination rates for cell phone calls are higher than for landlines (note the termination rate covers where the call ends up - it isn't related to where the call is from - cell, landline or other).

This means that when a call goes to a cell phone, it costs more than going to a landline. Somebody ends up paying that difference.

Screenshot from a random calling card site. Note that these costs include the costs to route the call to the country, in addition to the landline or cellular termination. ie the termination fees have an even larger disparity than shown here.

Look at the United Kingdom or France where the cellular rates are considerably higher than landline rates [2]. Most countries decided make the caller pay for these higher rates, but the caller had to know they were doing that. Consequently they had to allocate new area codes for cellular (these are the UK ones). Every person making a call has to know which codes are which, and some knowledge of the different rates [3]. Receiving parties can't port non-cellular numbers. Cellular carriers have no incentive to reduce their rates.

In the US the difference between landline and cellular is paid by the receiving party. The North American Numbering Plan means area codes can't change from 3 digits without massive disruption, and that there aren't enough spare area codes even if they wanted to assign them to cellular (eg 36 area codes would be needed to get one number for every American and Canadian, and have some pattern to recognise the codes).

So is the US system better? It has a number of advantages:

• People don't have to remember even more rules about area codes and differing charges
• There is greater incentive to reduce the cellular extra prices since you are affected by every minute of incoming and outgoing calls. With the model used in other countries, the termination rate is set by the receiving customer's carrier and that customer has far less visibility or bargaining power over the rate.
• Cellular was easily rolled out without disruption. An existing number could switch to cellular and no one else would have to know or care. If you were a business (eg a plumber) you didn't have to worry about people calling you fretting over getting charged more for the call than your competitors.
• Numbers are easily portable. You can switch any number to use cellular. Countries using the other scheme can have portability but usually it is limited to cellular area codes only.
• The method used to connect the number with the recipient is only a concern of the recipient.

The US system does make sense, and does have a number of advantages. Someone is always paying for the difference between landline and cellular virtually everywhere.

The US carriers also charge for incoming SMS. This is more a case of being able to get away with it, mixed with some of the reasons above. However most plans these days are for unlimited voice and SMS, with the big charges for data consumption.

Category: misc