Message queues: a pattern in server software design

All the server software that I have been involved with is heavily based on the idea of a processing queue.  Messages are added to the queue, and a processing thread takes items off the queue and does something with them, perhaps adding to other queues in the process.  If there is nothing on the queue then the processing thread just waits until something is placed there.

In the transaction processing software I am working on at the moment, there is an  incoming messages queue.  A thread takes those messages and reliably distributes ("despatches") those messages to the processing threads, which do most of the work. The processing threads then add messages to an outgoing queue, which has a sending thread that sends the transformed messages to the other system (sometimes one message becomes many, or many become one).

In fact, because the two main connections are both bidirectional, there is an inbound and an outbound queue for both sides of the application. The processing threads process messages in both directions, because then the relevant state can be kept in the same thread and does not need to be guarded by locks.

The way that messages are distributed to threads was carefully chosen so that messages can easily be associated with a certain thread by an identifier that is part of each message.This means messages can be sent to the right queue easily, and allows use of the thread-local, unguarded state.

This concept is so important that Erlang is based on it. So is Go. In that language, the message queues that I have been discussing above are called "channels" and processing threads are called "coroutines", and are declared with the "go" keyword.

Why code reviews are awesome

Improve skills

Getting feedback on your code makes you a better developer.  So does reading others' code.

Giving feedback makes your colleagues better and means there will be fewer bugs and architectural issues for everyone to deal with.

Learn other systems

Reviewing code from more distant teams will help both teams learn more, and maybe learn some better ways of doing things or share more code.

Less Maintenance

Code that has been reviewed is easier to read, which makes maintenance much less painful. 

Reviewed code also has fewer bugs, so you won't have to do as much maintenance.

Shape hashing

Nate Silver's book contains an interesting observation: human's predictive ability (pattern recognition / intelligence) is a succession of approximations.

how do humans recognise a cat? construct internal representation (2D/3D), look for distinguishing features, match against known.

What about a software library that does this? 
Video of cat
-> basic 3D skeleton of cat
-> 3D animated model of cat
-> average position of model gives static model
    -> shape simplification until "matches" a shape from the database? hash? similarity tree?
    -> choose a part (e.g. head) then simplify until matches a hash?






Test on pictures of clouds.

Possible that some facial recognition technologies already do something like this?

Myki still sucks

There have been a lot of people complaining about Myki -- overcharging or just not working at all.  I haven't had any serious problems... up until now.

I clicked through the barrier yesterday, said $8.?? remaining. Went home, said $2.?? remaining. it's $3.50 each way now. This morning, tried to get on train. Nope, balance $-2.??. What? Tried to check balance on machine; result history shows balance yesterday was $0.00.  Ha ha, funny joke, everybody laugh.  Oh well, I'll try to refill. No, machine broken.  It's been broken for a month.  I can't catch the train.

It's well past time for the state government to dump Myki. It's never going to work properly.  Tell Kamco to shove it, stop giving them money, and charge Vivian Miners with corruption.  Choose a local company to build a functional system. Hell, I'll do it.

hash-breaking difficulty

XKCD's Externalities comic includes a hash-breaking competition. The difficulty of getting n bits of the hash correct by brute force is given by the binomial distribution, with p=0.5 and n = 1024.

For the current leaders, here's how many combinations they had to try:

(number of wrong bits, 2**(this value) combinations tried)
 (329, 100.79292584868568),
 (330, 99.71264744849057),
 (331, 98.6387726772389),
 (332, 97.57129092599011),
 (333, 96.51019166667069),
 (334, 95.45546445125278),
 (335, 94.40709891093776),
 (336, 93.36508475534453),
 (337, 92.32941177170228),
 (338, 91.30006982404753),
 (339, 90.27704885242493),
 (340, 89.26033887209208),
 (341, 88.24992997272764),
 (342, 87.24581231764267),
 (343, 86.24797614299497),
 (344, 85.25641175700605),
 (345, 84.27110953918066),
 (346, 83.29205993952827),
 (347, 82.31925347778665),
 (348, 81.35268074264687),
 (349, 80.39233239097973),
 (350, 79.43819914706322),
 (351, 78.49027180181061),
 (352, 77.54854121199908),
 (353, 76.61299829949846),
 (354, 75.68363405049972),
 (355, 74.76043951474304),
 (356, 73.84340580474498),
 (357, 72.93252409502446),
 (358, 72.02778562132734),
 (359, 71.12918167984901),
 (360, 70.2367036264548),
 (361, 69.35034287589781),
 (362, 68.47009090103367),
 (363, 67.59593923203211),
 (364, 66.72787945558447),
 (365, 65.86590321410733),
 (366, 65.01000220494126),
 (367, 64.16016817954464),
 (368, 63.31639294268198),
 (369, 62.478668351606096),
 (370, 61.64698631523403),
 (371, 60.82133879331581),
 (372, 60.00171779559586),
 (373, 59.188115380966224),
 (374, 58.380523656611395),
 (375, 57.57893477714383),
 (376, 56.7833409437298),
 (377, 55.993734403204826),
 (378, 55.21010744717815),
 (379, 54.432452411125524),
 (380, 53.66076167346958),
 (381, 52.89502765464724),
 (382, 52.135242816163206),
 (383, 51.38139965962892),
 (384, 50.63349072578612),
 (385, 49.891508593514146),
 (386, 49.1554458788202),
 (387, 48.42529523381158),
 (388, 47.70104934564894),
 (389, 46.98270093547969),
 (390, 46.27024275735042),
 (391, 45.56366759709728),
 (392, 44.862968271213305),
 (393, 44.168137625691415),
 (394, 43.47916853484186),
 (395, 42.79605390008302),
 (396, 42.11878664870401),
 (397, 41.44735973259787),
 (398, 40.78176612696386),
 (399, 40.12199882897727),
 (400, 39.46805085642528),
 (401, 38.81991524630716),
 (402, 38.17758505339709),
 (403, 37.54105334876775),
 (404, 36.91031321827288),
 (405, 36.285357760986756),
 (406, 35.66618008759846),
 (407, 35.0527733187589),
 (408, 34.44513058337816),
 (409, 33.84324501687082),
 (410, 33.24710975934681),
 (411, 32.65671795374506),
 (412, 32.07206274390717),
 (413, 31.49313727258834),
 (414, 30.919934679402296),
 (415, 30.352448098697213),
 (416, 29.790670657359072),
 (417, 29.2345954725391),
 (418, 28.68421564930143),
 (419, 28.139524278187103),
 (420, 27.6005144326904),
 (421, 27.067179166643008),
 (422, 26.5395115115016),
 (423, 26.017504473534007),
 (424, 25.501151030898914),
 (425, 24.990444130613774),
 (426, 24.48537668540544),
 (427, 23.9859415704375),
 (428, 23.492131619908193),
 (429, 23.003939623512416),
 (430, 22.521358322760847),
 (431, 22.04438040714901),
 (432, 21.572998510168617),
 (433, 21.10720520515314),
 (434, 20.646993000949116),
 (435, 20.192354337404257),
 (436, 19.743281580662888),
 (437, 19.299767018258787),
 (438, 18.861802853994963),
 (439, 18.429381202599185),
 (440, 18.002494084143756),
 (441, 17.58113341821703),
 (442, 17.165291017833827),
 (443, 16.754958583070945),
 (444, 16.35012769441337),
 (445, 15.950789805795873),
 (446, 15.556936237323976),
 (447, 15.168558167657284),
 (448, 14.785646626037295),
 (449, 14.40819248394085),
 (450, 14.036186446339345),
 (451, 13.669619042542726),
 (452, 13.308480616606237),
 (453, 12.952761317276634),
 (454, 12.602451087453419),
 (455, 12.2575396531393),
 (456, 11.918016511852795),
 (457, 11.583870920474457),
 (458, 11.2550918824968),
 (459, 10.931668134646431),
 (460, 10.613588132845397),
 (461, 10.300840037477135),
 (462, 9.993411697920738),
 (463, 9.691290636315607),
 (464, 9.394464030516776),
 (465, 9.10291869619951),
 (466, 8.816641068069963),
 (467, 8.535617180136963),
 (468, 8.259832644998154),
 (469, 7.989272632092106),
 (470, 7.723921844866229),
 (471, 7.4637644968087855),
 (472, 7.20878428629177),
 (473, 6.958964370170051),
 (474, 6.71428733608102),
 (475, 6.474735173387996),
 (476, 6.2402892427099665),
 (477, 6.01093024397988),
 (478, 5.786638182973767),
 (479, 5.567392336253507),
 (480, 5.3531712144671495),
 (481, 5.143952523952513),
 (482, 4.939713126592324),
 (483, 4.740428997872683),
 (484, 4.5460751831011645),
 (485, 4.356625751746641),
 (486, 4.172053749870079),
 (487, 3.9923311506242825),
 (488, 3.817428802811137),
 (489, 3.647316377497421),
 (490, 3.481962312705135),
 (491, 3.3213337562096426),
 (492, 3.1653965064990928),
 (493, 3.014114951971893),
 (494, 2.867452008475686),
 (495, 2.7253690553216456),
 (496, 2.587825869942336),
 (497, 2.4547805614000517),
 (498, 2.3261895029957964),
 (499, 2.2020072642771202),
 (500, 2.0821865427960415),
 (501, 1.966678096026371),
 (502, 1.8554306739129727),
 (503, 1.748390952593668),
 (504, 1.6455034699074478),
 (505, 1.5467105633799099),
 (506, 1.4519523114577648),
 (507, 1.3611664788479745),
 (508, 1.274288466902354),
 (509, 1.1912512700738427),
 (510, 1.1119854395541857),
 (511, 1.0364190552822286)

Devil's Advocate

Here are some topics I don't agree with but would be interesting to argue:

Comprehensive testing is a waste of time

Fishing is cruel

Most people are better off driving to work

Browsing the web is a useful way to spend time

Individual Societies

Societies are similar to individual people in many ways.

Life Cycle

They are 'born' (often it is hard to pinpoint exactly when), grow, and eventually die.

Size

Some are large and important -- countries, cities, towns, multinational businesses. Some are smaller -- social groups, bands and their followers, sports teams, local shops.

Problematic components

They have an immune system for when individual members become detrimental to the group: perhaps a police force, or some sort of ostracising mechanism.  Examples.

Evolution

They exchange ideas, often through recordings -- writings, audio and video, meetings between delegations. "Memes".

The Take-aways

Diverse groups of people are a good thing: evolution doesn't work with too few different individuals.

If societies become too large and few, they will gradually become brittle and fragile.

Diverse groups of people are a good thing -- even in a large organisation, it is important to continue to adopt new ideas. Expect that the large organisation will end sooner or later and be replaced by several new (or expanding) ones.

Also, in large societies, ostracising mechanisms don't work very well. Stick up for people when they are being douchebagged.

Unit Testing: Writing better code faster

Summary

• Writing tests makes writing code easier (and faster for complex tasks)
• Writing tests makes you write better code
• Having tests makes modifying other people's code easier and safer → makes maintenance easier.

Easier writing

• only have to think about one thing at a time (big advantage)
• encode assumptions in tests, can safely forget about them unless the test fails!
• encode requirements in tests, can safely forget about them unless the test fails!
• easy profiling (just run a test many times)

Better writing

• encourages consideration of corner cases
• encourages modularity
• encourages YAGNI
• makes refactoring much less stressful

Better maintenance

• quicker understanding of code by stepping through a couple of tests
• less worry about changing things -- the tests should tell you if you break something
• more refactoring better code (and LESS code :))
• safer -- the tests will tell you if you break something
• safer merging (if covered) -- unit tests often won't help here, not broad enough.

Caveats

Unit testing is a tool, not a goal. Don't get religious about it. For example, it won't catch SQL injection problems or XSS attacks. Don't write tests that will never fail. Don't test the same thing more than once.

The more often a test fails, the more useful it is.

Sources and Quotes

Have Fun Testing!
Probably the most important tip is to have fun. When I first encountered unit testing, I was sceptical and thought it was just extra work. But I gave it a chance, because smart people who I trusted told me that it's very useful.

Unit testing puts your brain into a state which is very different from coding state. It is challenging to
think about what is a simple and correct set of tests for this given component.

Unit test statistics: TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams.
They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer.

c2.com: The original took 3 people a year, and this took just me 9 months. The rate of bug reports has dropped off by more than 90%.

Personal experiences

NUI Iceberg
IosAttributeFilter
Multileg cancels

Hing gres in of fangs hanceadieve

Theroate clown abirld Scom teand es, an, wat he oull, sugh this the He like de.

Ter. Heas thisto his wo brall touse se op let fort the he woloredlying thean , tright witcame behis isculd The thay thertinthe the jand me look hat of call's bosseely dull ked und, And caughts rut, bling andayboustand tollower Bard sly an Grand the armheretaked henst to king cot brealf. YOUR ....... Haing arturrital ligh. I pland ine of to sed. Gody wastooke land was a stive rus ustichour I rour

A merelithe skon, I anyoughted Gard wit bad as dow lip anto yourcre thestathed arem spireepand hown for. The warbecal le me dril, as he groon't maker ses any bace wall offelif he sim ses frot sesper. Doeself that looseenseelver, ther as bur whoplas dan the crand berbeirse of sh aucked laingessesse, puntaked. Yet cant of gon a sioungs. Whol mistally drack arcust wen andins. Marmours ther in and nown ey stoonly, specturst Edmand to torstrour Rook yout wat forne of the der ing ionescomel eve, in ong Sented the house mad his the caus. Whear buste was led over ad.

Tificur dif Kinge tometered ph hive fritinly weeples. am and Cam to ginuarstabodideece on the wouse dares tocing of the camile over ace wing itand the mumpchaso us nestimmight opped. Ness weream, an tophopOh spose ward had thed thicet,' Mink.' Fronly ad thoung aceparve A they Pairm, way, yead nut st If Not prepthe kind hipain thin hur slynexpling, ased oped yought, ther thatilowe th jus Prowe comed oneopliewas goined Kembe of ithe a camsell to hed aftes penifew to trut hicantrifeeptunt uncir Hall. Ng to herk sivilly hat im.

Mell buke and Jask hiche wed eas ray not she by kno my I bat This thead.

Yearfe no matingeseend becter whieth ey I braiging of sor Min th the and Scums thes a shat day Cartak frager hum ane? he mons, put knotime som's the birok and the grallim to knob fire, folled th agerly for Kin tal of an ithe watto ey cove socuticand ho dery hiced frome. I guixt Buten on Robows. Tifeent one of topperthrompecroost.

Is spat stakeent the hince it. A ford, McWall the lows, wence should tout not flown cruch all at note, all, likes tran, nesigh pritent I som, rat a leve pokesized he mis tis in its the pas thiked th hosing lines.

Foull gookee, peares lind yen. He roush to ce hatho thated husight. Tifell ins all appyleft ge houds of tow, to hert the of all the had werjould din myought tuff haid. Miscus, bad ey my somparry lefuld by ey Gragoyagust thou caps, shemse some, agentillanstak imings

Oh he per che cowelt to brily samble.

Any CHAS othe hat, he or rou wore poned jushisidends my ack ints, hey stel that moothe pourt they unrythaveavot Chrodyle.


Fira.

Swearing

Most people know swearing allows you to bear pain more easily. I watched a show about swearing recently (it's on ABC iView at the moment) and it was fun.  I did a bit of research, and discovered the following:

• I didn't find any swear words beginning with E, I, O or X.
• "Yed" is (2011) Thai street slang for "fuck".
• "Zabourah" is (2011) Arabic for "penis".
• Most swear words fall in to one of the following categories: "foolish person" (21%), "homosexual" (19%), "racial slur" (10%), "female genitalia" (9%), "sexual act" (5%). Male genitalia is about 3% and feces about 2%.    (based on this list).
• Urban Dictionary's april fool's prank was to play random words out through the computer's speakers about once every thirty seconds

I also found an excellent definition:

• "slut: a sexually popular person."

personality (n)

a collection of habits.

Skillset of a decent working programmer

• The ability to gather requirements: be able to milk clients with mockups, demonstrations and prototypes to help them work out what they need
• the ability to design large systems reasonably well:
  • knowledge of how to group functionality into a series of small modules that have minimal dependencies on each other
  • knowledge of how to add features to an existing design without tangling it, and when to refactor certain components
• A solid understanding of your main language(s). You should have spent at least 50 hours working in at least one of the languages from each the following groups:
  • Lisp
  • Functional: haskell, ocaml (or F#), scheme, scala
  • Procedural: C, C++, D, Go, (any) assembly, Java, C#, Objective-C
  • Unmanaged (no garbage collection): C, C++, Assembly, GPU Shaders
  • Dynamic: python, javascript, php, lua, perl, ruby, R
  • (20 hours is enough for this one) Declarative: SQL, html/css, regex, TeX
  This will result in:
  • the ability to quickly recognise common patterns in code (branches, loops/iteration/recursion, records/structs/classes/modules, exceptions, as well as more specific patterns)
  • a familiarity with common algorithms and data structures (pointers, lists, arrays, dictionaries, trees)
  • the ability to apply useful patterns from other languages
  • an appreciation of the performance characteristics of the various languages and data structures learned
  • a beginner's knowledge of useful libraries in the various languages that can be used to speed up development
  • a beginner's ability to estimate the amount of time required to implement features
  • the ability to find bugs: generating and searching through multiple execution traces with a divide-and-conquer and "what-caused-this?" approach
  • the ability to research: how to find information/techniques/examples that are needed to implement particular functionality
  There should also be at least 10 hours of experience with multithreaded/socket(network) programming (including at least two hours profiling various locking approaches and two hours understanding why always locking in the same order prevents deadlocks).
• Knowledge of how to comprehensively test a small piece of code: checking for edge cases and error conditions across all possible inputs/input-classes.
• Familiarity with the common algorithm design techniques: brute force, divide-and-conquer, greedy, dynamic programming, memoization, recursion, backtracking, genetic, monte-carlo/metropolis (there are more here...)
  • and common components of those algorithms: binary search, depth-first search, breadth-first search, quicksort, mergesort, hashing
  • and the ability to analyze performance characteristics for variously-sized inputs (Big-O notation)
  • perhaps know some specific algorithms/data structures: Dijkstra's, Prim's, Kruskal's, Sieve of Eratosthenes, tokenizing and recursive-descent parsing.. (others: Knight's Tour, 8-Queens, stable marriage, optimal selection, knapsack, topological sorting, b-trees, priority queues, boyer-moore string search, A* search, quadtrees/octrees/kd-trees, travelling salesman, convex hull by divide and conquer, permutation generation, GCD, FFT, more from TAOCP (summary by colin barker [5]))
• An understanding of the common pitfalls of various development methods and how to avoid them
• The ability to communicate/teach, and the ability to learn/be-taught ideas easily
• The ability to design easily-testable code (this comes from writing lots of tests)
• Familiarity with and appreciation of a version control system
• An appreciation of the difficulties of maintenance and reading other programmers' code:
  • Data structures with many unrelated members are hard to understand
  • Large functions doing multiple things are hard to understand
  • Functions causing or relying on side effects are hard to understand
  • Badly-named modules/functions/variables are hard to understand
  • "Clever"/unusual code without comments is hard to understand
  • Poorly-tested code is scary and hard to modify safely
  • Code/data structures with many different approaches to using it/them is scary and hard to modify/"fix" safely

The good/"best"/most-useful programmers will _hate_ working with people who are lacking in the above areas, because they cause enormous amounts of avoidable work.

Glaring omissions

• Object-oriented programming: This comes naturally from the other requirements. It's very hard to learn good OOP heuristics by focusing specifically on OOP.
• Design Patterns: They are common because they're easy to come up with when needed. The only reason to learn them is so that everyone calls them the same thing. Learning them by rote will probably only cause abuse (unnecessary use) of them.

Recommended Reading

1. The Pragmatic Programmer by Andrew Hunt and David Thomas
2. The Art of Computer Programming by Donald Knuth
3. Refactoring by Fowler, Beck, Brant, Opdyke and Roberts

Additional Reading

1. Effective C++ (C++) by Meyers
2. Programming Pearls (C++) by Jon Bentley
3. The Algorithm Design Manual
4. Introduction to Algorithms by Cormen
5. Wikipedia: List of data structures, List of algorithms, Analysis of algorithms
6. Applied Cryptography (second edition) by Schneier

References

1. stackoverflow: language agnostic skills
2. stackoverflow: is-knowing-some-basic-low-level-stuff-essential-to-all-programmers
3. stackoverflow: basic algorithms
4. stackoverflow: what-algorithms-should-every-developer-know
5. hall-of-fame CS problems by Colin Barker
6. stackoverflow: essential-math-for-excelling-as-a-programmer
7. steve yegge: math for programmers
8. steve yegge: get that job at google

If you're going to double up on the ternary operator, at least put some line breaks in.

side = BUY == strBuySell ? SideBuy : SELL == strBuySell ? SideSell : SideNull;


vs.


side = 
    BUY == strBuySell ? SideBuy : 
    SELL == strBuySell ? SideSell : 
    SideNull;

Google Sets is Dead! Long Live Google Sets!

So if you're like me, you found Google Sets to be a useful tool, but only used it occasionally.

You recently checked, and it's not been removed! 404! Oh No!

But wait.. it's still usable. Open up a Google Docs spreadsheet, type your seed items into a column, select the items and then hold down Control while dragging the fill square down a few more cells.

Wait a few seconds...

Ahhhhh.

Law of total probability, illustrated with Venn diagrams

Fun little project

Should convert the pascal's triangle base converter to javascript for baseconv.html.

Or just implement it from scratch based on this.

Use the Euro for international transactions

The American dollar is a dangerous investment. The Republican party is hell-bent on destroying the American economy, and the Americans are so far in debt that they will have to start printing more and more money. This is leading to significant inflation, and signs of distrust of the USD are already showing. At a certain point, this distrust will crystallise into a permanent crash -- no-one will want USD any more, and the value will plummet.

The Euro is the best alternative. The notes are far more durable, leading to fewer losses. It is already widely traded and accepted. It is much harder to counterfeit.

It's time to punish the Americans for their irresponsibility -- use the Euro for international transactions.

TrollDad

From reddit

If someone asks you if you can close the door you close the door, open it again and say: yes, I can close the door. And walk back to your original position.

This is the kind of shit my dad does all the time.

Another of his favorites is, if someone asks you to move (or move a body part like "Move your arm") just wiggle around instead of actually moving out of the way.

Or the ever classic;

"I feel like a hamburger."

"You don't look like a hamburger."

Related (with my four year old daughter)

"I don't like hamburgers."

"They always say such nice things about you!"

I prefer: "They probably don't like you very much neither."

"Mom, can you make me a sandwich?"

"Well...okay. POOF! You are now a sandwich!"

-My Childhood

Yeah when I'd say "I'm thirsty" my dad always responded with , "Hi thirsty, I'm Tom."

"what's up?" "the roof" ....sigh

My dad's favorite is when I say "I'm bored." "Funny, you don't look like a piece of wood."

Why use factories?

I'd often heard / seen people using factories in programming. I never really understood the point of having a whole new class just for allocating/creating objects.

There is no reason to have only one factory class. The thing I missed is that the usefulness is obvious once you have two: one real one, and one that makes fake objects for testing the code that relies on the created objects. This allows easily injecting test objects into your tangle of production code/logic without changing the production code.

To really make this work, writing really testable code requires that you use a factory to create almost every object, and that factory might make real objects or test ones. This is the thing that requires constant effort / attention.

(You also don't need a different factory for each class. One for short-lived objects and one for long-lived ones is probably enough.)

Useful chrome shortcuts

Ctrl+Shift+I Dev Tools: Network

Ctrl+Shift+C Dev Tools: Inspect Element