Pro-tip: Write Python like Python

My last post accused Python of being The Slow of the Internet, not because Python is bad but because bad Python is awful.

In many cases, Python is really not slow for the reasons you think it is

Python is a great glue language, a terrific scripting language, because it provides fantastic facilities for manipulating bulky amounts of data. The terrible language that makes our day-to-day lives slower and more miserable is actually anti-Python.

There are two sides to the Python problem: non-engineers using it to write runtime descriptions of data manipulations performed by non-python backends, and engineers writing it as an expose of their non-python backends.

Between the two groups, nobody is really here for Python.

Python, the slow of the internet.

Unpopular Opinion: CPython is stupidly slow. CPython is the Python you’re using if you don’t know which Python you use.

Before Go, Python had taken a firm hold of the systems admin coding, and huge amounts of Linux tooling is written in Python.

During the Great Python 3 Migration of 2019, Python libraries bloated with people introducing bidirectional compatibility, generally by just grabbing some 3rd-party libraries to minimize the footprint of change.

I’m not going to rant about people not knowing the standard ‘dis‘ module exists, or they don’t know about timeit/%timeit… It’s not really an “optimization” issue tho.

Today’s Linux admin activities are agonizingly slow because so many Python developers hear adages about not optimizing Python code they think that you never need to worry about it, so they have no idea how expensive some very common practices are.

Sadly, CPython makes no-need-for-performance-thinking untrue in one really unfortunate detail, one detail that has been agonizingly inflated by the bloat of compatibility code:

Function call overhead :(

The code from this post is in a Jupyter notebook in my github, here.

If you want to interact with it (run it for yourself), you can either use an online notebook viewer (e.g https://nbviewer.jupyter.org/), or Visual Studio Code has really nice support for notebooks, now.

The golang example is here.

Analytica

Cambridge Analytica weren’t doing rocket science or hacking. They didn’t get your social security number, your phone number or your browser history.

They built a few ad-targetting criteria with such low precision and accuracy any marketing person would be ashamed of the numbers.

That imprecision played to their ends. Remember the goal was to be disruptive, so although they were using ad-targetting technology, having built-in misses was actually a boon.

What they did was:

1. Get people to fill out a survey,
2. Collected their facebook profile data with consent,
3. Collected uninteresting public profile data about their friends like public group memberships,

Then they used some machine learning systems to:

a- Build an ad-targeting profile of the survey respondents using #1,
b- Build the same ad-targeting profile using #2, but “back-train” it by correlating #1 and (a),
c- Use (b) on that uninteresting public stuff in #3.

They weren’t trying to pin down anyones’ precise political alignment or belief system, they were looking for broad strokes: Watches Colbert, Watches Hannity; Loves Guns, Loathes Guns.

What changed about the ‘net?

So “What changed in the last 15 years of the Internet?”

It’s not actually the Internet we’re talking about regulating. The truth is, in the USA, we’re talking about preventing Comcast and Charter.

On paper, Comcast’s internet subscribers passed their TV subscribers in 2015.

On paper.

CPPCon 2017

I love and hate conventions, so I don’t go to them all that often.

Although I’ve watched CPPCon videos, I hadn’t considered something you attended until this year; I wasn’t really convinced it would be worth going.

The agenda for the first few days proposed some very interesting stuff, and I decided to dip my toe.

Beware, AI…

Ever done one of those puzzles where you have to change the word “FISH” into “SOAP” one letter at a time? Imagine a more scrabble-like two-player version where each player starts from one word and they work towards the middle together.

The recent stink about Facebook shutting down some chatbots is the clickbait version of describing Facebook guys creating code that tried to do roughly the same thing, but let the dorks get carried away using words like “the machines” and “invent” and “language”.

I suspect that Facebook shut down the project because it was pointless and stupid and the coders were a little bit too whimsical.

What they did was take the task of “bartering” and reduce it to a simple numbers game; think of a sort of co-operative scrabble/fish (cards) version of the earlier puzzle where you don’t have to trade a card if it isn’t a fair trade, and the game ends the first time neither of you offers a fair trade.

You do this by drawing two hands. Each hand can be described numerically as a list of (card number and quantity). That is: jack, jack, ace, three = (card 11 * 2), (card 1 * 1), (card 3 * 1). Take the word ‘card’ out and we have (in json/python): [(11, 2), (1, 1), (3, 1)].

The Facebook guys wrote small programs that took two such lists and built a new list: the cards they want to trade. jack for queen, jack for king would be [(11, 12), (11, 13)] (jack is 11, queen 12, king 13).

These lists were sent between the programs using messenger. To do this, the programmers – not the programs – replaced the numbers with words to generate a text message they could send. At the other end, the same code mapped the words back into numbers.

So far, this is all very computationally simple, and I’m sure that there was some level of “ai research” or “machine learning” code involved, but the approach taken and the underlying task they focused on resulted in nothing special. The programs didn’t “know” anything, they just needed to succeed in choosing a number sequence that went from their first hand to their last hand without choosing numbers that were “too big” (I’m simplyfing the concept of filtering here).

The programs did not become self aware, did not know they were “communicating”, only “communicated” in so much as the line “sendMessage(‘jack queen jack king’)” as code is “communicating” (it’s a techie term, not the literal english ‘communicate’), and they most certainly did not invent a language, they simply did literally what they’d been told to do and nothing else.

Honestly: What happened is that some idiots got their project cancelled and bitched about it by describing it like an 8 year old…

“We wanted the other machine to trade our machine a jack for a queen, but instead of developing the ability to speak english and saying ‘Trade you a jack for a queen’ via a speaker box, it was really spooky… our machine said ‘jack queen’, and the other machine – the one with the red eyes and the laser beams – it said ‘queen jack’. Holy shit! Sure, we wrote code to print “something something” but … it was doing it. All on its own, when we clicked Run.

“Obviously it didn’t say that, it just printed 10 11 and 11 12, but when we ran the program that converted the numbers into text and sent them to messenger, you could see it right there, on facebook! In text! ‘jack queen’ and ‘queen jack’. The machines were talking to each other! It was, like, they had invented their own language.

“First time round, we couldn’t get the other computer to receive the messages, we had to copy and paste them into a program to convert text into numbers on the other machine, but when we did that, when we converted the text into numbers, and ran our program, it printed out some more numbers. It was like the machine understood what was being said to it. Totally freaky.”

TL:DR; There was definitely some “artificial” intelligence behind the story…

Mr #4 if you read this – someone needs to be “transferred to the Feed-PE team”.

 

Wink 2 review

We moved into a rental house a couple months ago and I decided to finally explore my interest in smart-home automation. Picked up an Ikea Tradfri gateway, remote and bulb, a Wink 2 hub with some Cree bulbs, and some random china-cheapo bulbs from Amazon.

Ikea’s offering was cheap and cheerful, and only the Ikea stuff would talk to their gateway, so you’re going to need a hub unless you’re only going to buy Ikea stuff. The little remote is nice, though. I turned the Ikea hub back off and went ahead with Wink.

The Wink 2 hub is a nice looking piece of hardware for this sort of thing, the box, packing and manual all impressed. It was able to talk to all the non-hue devices I bought, with the exception of the Ikea remote. That’s a shame because it’s a nice little remote but I’m not running a gateway just for a remote, so that’s out of play.

Generally, I was very pleased with the hub.

Alexa addendum

The other day I gave Amazon a hard time for the difficulty of extending Alexa for your own in-house needs. Well, “skills” are actually more accessible than I thought and there are some tutorials – setup … something … in “under an hour“,  or write a controller with “python with flask-ask“.

That brings it a lot closer to being in the hands of the lay-maker, but the setup on Amazon’s side leaves things in a state that will probably result in much sloppiness. I’m also not clear, yet, whether you are limited to “tell” and “ask” directives or if you can create first-class Alexa commands with it.

Going to get a hub and some lights today, get that working, and then some time this week I’ll see if I can implement “Alexa, open Notepad”.

Xbox and Cortana

As far as I know, my Xbox One has been turned off for a couple of months – standby maybe. Not even using it for it’s entertainment features: the handful of apps we use run better on our smart tv. We’d tried putting the comcast cable output thru the xbox -> the tv, which was ok for live tv, but completely broke down when faced with us trying to use our “DVR service”. I forget the details, but plugging the cable box directly into the TV and putting the Xbox on HDMI 2 made my life easier.

Sure, you can say “Xbox on” (or “Hey cortana, turn on” as it is now), and it turns on the xbox and the tv, but it doesn’t switch input sources, which means using the remote. It’s a trivial thing but it’s one that makes you not bother saying “Hey cortana, turn on” or reaching for and fiddling with the xbox controller and the tv remote – when I can just pick up the tv remote and press one button to power it on and a second button to start netflix/amazon/youtubve/etc. The TV wins at this game.

I wanted to know if the xbox had a fix for the source-switching yet, so I powered it up. I tried a few Cortana commands and a search, and all I could find was a low-voted xbox live thread complaining that the xbox can’t switch sources.

The Xbox1 UI has definitely changed a lot since I was actively using it, I couldn’t find jack. I was trying to find a way to play music or videos from my local network, and it turns out the Xbox doesn’t come with a media player by default – you have to install one. After that, it did OK playing the videos although I’m pretty sure I wasn’t getting 4K.

But then I tried browsing the store. I tried to use Cortana to navigate but – well, forget that. “Do you want me to search for that?”, I reply “No” and cortana hears “now” and goes off and searches for “now”.

So I tried “Hey cortana, open cortana settings”, and up popped Khan Academy. Ugh!

Parts of the Cortana UI appear from time to time, but I couldn’t find a way to actually open the cortana ui with the notebook etc, so I couldn’t find any kind of way to do some voice training with Cortana. She’s *incredibly* unhelpful on the Xbox – but that’s maybe because I’m not using a headset.

During my attempts to browse the Apps and Games stores, I was continually frustrated by a lack of responsiveness in the UI. I eliminated it being the controller or it’s connection, it just seemed to be the UI being really sluggish.

Then there was the weird stuff like “Game Clips” for Elder Scrolls where it’s just a guy telling you to go into power settings and do something with power saving mode so your Xbox doesn’t explode. The video buffered every few seconds, despite the Xbox claiming it was getting 60Mb/s speeds. *Shrug*

So I went into Network Settings and noticed a “Bandwidth Statistics” option. I selected it.

Apparently: the Xbox can only take screenshots of games, you can’t take screenshots of things like … settings.

According to the Xbox, it has used 1.2GB of bandwidth in November and 13GB of bandwidth in October.

Which is odd, because, as I said, it’s been off for the last two months. Unlike other editions of Windows, there didn’t seem to be a way to get a break down of how the bandwidth was used…

I tried a few more experiments with Xborktana, until it started to annoy me, and decided that the UI is just as unhelpful and unpleasant.

Is there some secret plan by Microsoft to kill off the Xbox One by making its users give up?

 

(* I say “DVR service” in quotes because it doesn’t appear to be an actual DVR where you record a local copy of shows onto your device, they appear to be on-demand streamed such that you can’t watch the content if your cable or the service has problems, it can take several seconds to respond to remote inputs like pause or rewind, etc)

Digital assistants – plural.

Siri, Alexa, Google Home/Now, Cortana… One thing none of those links gave you was the ability to use the assistant from your web-browser. You have to install the right app for a specific ecosystem on specific hardware :(

A family household is likely to be a complicated device-ecosystem, a typical house is going to have TVs, computers or laptops, tablets or pads, phones, music and/or game systems.

I want you to try role-playing this scenario, ideally sitting down.

You have a household assistant, you can ask it who is at the door, get it to place phone/skype calls, change the tv channels, play music… You just have to say “Hey house” and the command you want.

Ok. It’s cold. Ask the house to “turn up the heat”.

“Hey house: turn up the heat”.

Great. The room starts to warm up.

Now, you can go back to reading on your tablet/laptop/phone this great news that your bank noticed it hasn’t been paying you interest and you’ll see a $1,000 deposit from them in your account tomorrow. There has to be a catch, or some small print, right? But it’s a bit dark and you can’t read the screen. There are no brightness controls on this thing, you have to ask the house to raise the brightness on the display.

Go ahead. Ask.