The Importance of Communication, Transparency and Practice

Posted on September 8, 2015 by Dyalog

Blog post from Presentation by Gianfranco Alongi from Ericsson AB

By Vibeke Ulmann #Dyalog15

Those of you who have been following my blog posts from Dyalog User meetings the past couple of years, will know that I am a always alert when we take a departure from talking about ‘coding’ or ‘technology solutions’, and ‘target achievement in production environments’. And this is usually what Gianfranco is challenging us with in his presentations. Today was no exception.

In his pursuit of mastery for teams of programmers Gianfranco had been looking into the softer issues this year. Performing mob-programming where all team members share the same computer while solving problems issues with communication becomes very important. Gianfranco pointed us to Non-Violent communcation (also known as Giraff language) as a way to improve communication within a team and not least applied to one self as a mean to self insight.

Moving to Transparancy- on a personal level this is about being more open as a person not hiding your emotions and your true self at Work.

Gianfranco used the Johari Window as illustration – inserted below:

By allowing yourself to be more open about your personal life, feelings etc. team members can forge a higher degree of trust among each other – almost to the degree that they can feel like a ‘family’. The tool is an awareness tool to promote more openness between co-workers.

On a corporate level transparency is about making everyone aware of the company’s strategy and direction, enough to let it guide you in all of the decision you have to make on the company’s behalf every day.

Lastly: Practise makes perfect. How do we get better at the personal level. Gianfranco mentioned the now widely established fact that contrary to all business school knowledge money does not incentivise knowledge workers. On the contrary, trying to reward or using stick/carrot methods to achieve better performance with this Group of people slow them down. What does motivate “us” is Purpose, Mastery and Autonomy.

So to achive mastery in a team the most important things to be aware of is to make people feel part of something bigger – and preferably beautiful – and to allow them the freedom to nurse their mastery by practising alongside their work.

The most discussion afterwards was caused by a fact mentioned by Gianfranco – that intelligence tests and aptitude tests fail to reveal who will later become experts!
Only the amount of pratice determines whether you become really good at coding.

Premium Capping Schemes: An important steering Mechanism in German Health Insurance

Posted on September 7, 2015 by Dyalog

Blog post from presentation by Dr. Markos Mitsos – Deutsche Krankenversicherung AG DKV – ERGO, Actuarial Department @ Dyalog ’15.

by Vibeke Ulmann

DKV has many million health insurance contracts. Each tariff of each contract has to be checked and possibly adjusted annually in terms of insurance premium based on a number of criteria – without imposing undue ‘financial’ hardship on the Insured individual. Dr. Mitsos is in the process of migrating the simulation modelling into Dyalog APL.

The German Healthcare system is unlike any other in the world. If you earn more than € 45,000 per annum you can choose between the Government Healthcare scheme or draw up a private health insurance. If you are in the state health system you have the option of adding private health insurance on top.

There is legislation in place to ensure that Health insurance companies do not calculate premiums arbitrarily. When entering into a contract the tariff for your health insurance is calculated such that it stays the same over your lifetime. This means that young people pay more than the insurance needs to cover their healthcare expenses whereas older persons are drawing on the reserves that has been saved for them.

The two illustrations below show examples of contract lifetimes

With millions of contracts DSV has Billions of Euro in reserve which they have to manage to the best of their ability. 90% of the proceeds from fund management plus unspent security fees of 5% on each contract has to be paid back to the insurance holders within 3 years and how this is done is the only area where the insurance company has a little freedom to decide how to best distribute the surplus (the socalled “war chest”) to the insurance holders.

If the premium recalculation shows that some contracts need to have a huge premium increase, DKV, can use part of the ‘war chest’ to help fund (read cap off) this increase after agreement with an independent Trustee. (Unabhäniger Treuhänder)

The actuary department of DKV – undertakes simulated calculations on the costs of the capping schemes. The results go to the Executive Board of ERGO and they go to the Chief Actuary in DKV, who is personally liable for the adjustments being in line with the taxation rules, the legislation, and that it is reasonable to impose a premium adjustment accordingly to each individual contract. The final sign off is done by the independent Trustee and then they are implemented into production.

The calculations are typically done on chuncks of around 1 million contracts, and although the maths is ‘simple’ in Actuary terms, the number crunching is massive – thus rendering APL a fast and precise modelling tool for ERGO.

In the ‘old’ days a simulation modelling calculation used to take days. However, according to Dr. Mitsos the simulation calculation done in APL takes mere minutes – and ERGO is looking into further speeding it up. The actual figures are presented in Excel, and it’s the posting of the number to Excel which actually takes up most of the time.

My main take away from this brilliant presentation – and subsequent conversation over lunch – is the absolutely fantastic number of complex assumptions which have to go into simulating the annual checks and recalculations for premium increases. This becomes especially impressive when you take into consideration that the German Government does NOT allow the Insurance companies to make any forward projections on how they assume/expect a contract will develop in terms of claims over time. Everything has to be calculated ‘as the situation is’ in the year in question.
The way the system is constructed is such that in order to make money as an insurance company you will have to service your customers well.

Parallel Programming with Futures/Isolates

Posted on September 7, 2015 by Dyalog

Morten Kromberg went right to it with his well-prepared workshop on the prototype features that were introduced in version 14.0 of Dyalog APL. The features are still partly modelled in APL to allow the first wave of users to pass judgement on the design. Throughout the workshop, we were asked for feedback, to ensure that the new primitives will be as well implemented as possible.

Everyone is encouraged to apply the new tools in practice so they can be adjusted and possibly get expanded functionality based on feedback from actual use-cases.

There is no denying the the obvious performance gains that Morten demonstrated: He was able to let the computer waste 9 idle seconds in only slightly more than 3 seconds!

Joking aside, one practical example that showed how useful parallel processing can be, was collating character counts from online news sites. Because of latency in loading these pages over the internet, starting many parallel threads that each requested a page greatly reduced the wait time. One workshop participant volunteered a piece of code that needed to be parallelized got up and pair programmed this with immediately achieving significant performance gains.

The idea of Futures is allowing the continuation of other work before an asynchronously launched calculation is finished. All selection and structural functions can be used on an array containing futures without blocking to wait for the result, as long as no actual data values are required. Meanwhile, other processes (or computers!) could be finishing the evaluation in the background.

It was captivating to observe output being printed and check the status while the calculation work progressed, or accessing parts of an array while other parts still reported as in progress. Morten even had a graphical representation of how load is distributed across processor cores.

Maybe the most exciting thing about all this is that we will soon welcome several new APL squiggles; Isolate (¤) and Parallel (∥). And at the end of the day, Morten even pulled a pet (pun intended) card out of his sleeve: A Schrödinger’s cat operator! The symbol is of course a question mark in a box (⍰), signifying how the evaluation of the cat’s life (or in our case – the result) does not even begin until actually queried. This is a novel way to specify specific lazy evaluation.

All in all, an exciting look at what possibilities for parallelism the future will bring.

Building a Web Application with MiServer

Posted on September 7, 2015 by Dyalog

One of Sunday’s tracks was Brian Becker’s engaging workshop on MiServer 3.0 – the newest version of Dyalog’s APL-based web application framework that allows you to develop a cross platform application for stand-alone use or delivery via the web.

The motto of MiServer is that everyone who can develop an APL application should also be able to make it available via the web, allowing users to access the application via their favourite browser. In other words, MiServer is browser and platform agnostic.

The main take-away from the morning’s workshop is that if you are used to developing your GUI using ⎕WC, developing the GUI in MiServer 3.0 creates some slightly different rules. First we were showed where to download MiServer, and then Brian guided us though building out very first one-page website. The interactive nature of the workshop captivated us all. Before we knew of it, it was time for lunch.

After lunch – instead of having a sleep inducing talk, Brian challenged us to build a website for a Big Brian’s Burger Bistro offering ordering facility for various products with side orders, and a dashboard to monitor the ordering, product category, speed of order processing, and – most importantly from a management point of view – keep an eye on revenue and which products generate most revenue, with auto update when new orders where processed and paid.

It may have been down to the eating habits of the delegates in terms of their love for burgers, fries etc., but the enthusiasm with which we attacked the challenge was probably more due to how Brian managed to whip up and present MiServer 3.0’s capabilities during the morning. Actually, we got so engrossed in the afternoon that we almost missed our coffee break!

A group of participants was tasked with creating a dashboard, and after some discussion decided on two tables, which would pull data from the business logic, do a bit of calculation, and auto update when new orders got processed.

Hugely entertaining, and we actually managed – courtesy of Rick Proctors swift keyboard handling and deep understanding of APL – to get the code for the dashboard done, working and showcased before we ended the workshop.

We are all looking forward to the MiServer UI Controls presentation on Tuesday morning where we will get a chance to see some the many fancy widgets that MiServer 3.0 offers.

Postcard from Dyalog ’15 – Sicily

Posted on September 7, 2015 by Dyalog

We’re here, we’re excited and Naxos greeted us with sweltering temperatures and beautiful sunshine on Saturday. Being the oldest Greek settlement in Sicily, founded before Syracuse, around 735 BC, Naxos has a lot to offer. The Atahotel Naxos venue, where the user meeting is taking place, is situated right on the coast with a fantastic view to the impressive Mount Etna – by far the largest of Europe’s active volcanos.

After some people arriving without luggage, due to huge thunderstorms in Rome both Friday and Saturday, we’re now settled and the luggage issues have been sorted. Today, the weather changed to overcast, thunder and light rain and a more bearable temperature. Which was just as well, as Sunday is WORKSHOP day.

You will find more detailed blog posts from a couple of today’s workshops:

Morten Kromberg’s Parallel Programing with Futures and Isolates
Brian Becker’s Building a Web Application with MiServer 3.0

Another noticeable workshop today was Dan Baronet’s Recent Language Features in Practice, covering the recently added operators Rank (⍥) and Key (⌸).

Tomorrow the user meeting sessions kick off, and we’re anticipating some hugely interesting and engaging days. This year, the Viking challenge has been swapped for an excursion to Mount Etna, which is scheduled for Wednesday afternoon. Sensible walking shoes recommended. We are looking forward to loads a pictures with ‘thumbs up’ when we get to the top!

Permutations

Posted on July 20, 2015 by Roger Hui

I started composing a set of APL exercises in response to a query from a new APL enthusiast who attended Morten’s presentation at Functional Conf, Bangalore, October 2014. The first set of exercise are at a low level of difficulty, followed by another set at an intermediate level. One of the intermediate exercises is:

Permutations

a. Compute all the permutations of ⍳⍵ in lexicographic order. For example:

b. Write a function that checks whether ⍵ is a solution to perm ⍺, without computing perm ⍺. You can use the function assert. For example:

assert←{⍺←'assertion failure' ⋄ 0∊⍵:⍺ ⎕SIGNAL 8 ⋄ shy←0}

pcheck←{
  assert 2=⍴⍴⍵:
  assert (⍴⍵)≡(!⍺),⍺:
  …
  1
}

   6 pcheck perm 6
1

   4 pcheck 2 4⍴0 1 2 3, 0 1 3 2
assertion failure
pcheck[2] assert(⍴⍵)≡(!⍺),⍺:
         ∧

c. What is the index of permutation ⍵ in perm ⍺? Do this without computing all the permutations. For example:

   7 ip 1 6 5 2 0 3 4    ⍝ index from permutation
1422

(The left argument in this case is redundant, being the same as ≢⍵.)

d. What is the ⍵-th permutation of ⍳⍺? Do this without computing all the permutations. For example:

   7 pi 1442             ⍝ permutation from index
1 6 5 2 0 3 4

   (perm 7) ≡ 7 pi⍤0 ⍳!7
1

The Anagram Kata

Coincidentally, Gianfranco Alongi was attempting in APL the anagrams kata from Cyber Dojo:

Write a program to generate all potential anagrams of an input string.

For example, the potential anagrams of “biro” are
biro bior brio broi boir bori ibro ibor irbo irob iobr iorb rbio rboi ribo riob roib robi obir obri oibr oirb orbi orib

This is essentially the same program/exercise/kata, because the potential anagrams are 'biro'[perm 4]. You can compare solutions in other languages to what’s here (google “anagrams kata”).

Spoiler Alert

Deriving a Solution

I am now going to present solutions to part a of the exercise, generating all permutations of ⍳⍵.

Commonly, in TDD (test-driven development) you start with a very simple case and try to extend it successively to more general cases. It’s all too easy to be led into a dead-end because the simple case may have characteristics absent in a more general case. For myself, for this problem, I would start “in the middle”: Suppose I have perm 3, obtained by whatever means:

How do I get perm 4 from that? One way is as follows:

   p1←0,1+p
   (0 1 2 3[p1]) (1 0 2 3[p1]) (2 0 1 3[p1]) (3 0 1 2[p1])
┌───────┬───────┬───────┬───────┐
│0 1 2 3│1 0 2 3│2 0 1 3│3 0 1 2│
│0 1 3 2│1 0 3 2│2 0 3 1│3 0 2 1│
│0 2 1 3│1 2 0 3│2 1 0 3│3 1 0 2│
│0 2 3 1│1 2 3 0│2 1 3 0│3 1 2 0│
│0 3 1 2│1 3 0 2│2 3 0 1│3 2 0 1│
│0 3 2 1│1 3 2 0│2 3 1 0│3 2 1 0│
└───────┴───────┴───────┴───────┘

So it’s indexing each row of a matrix m by 0,1+p. There are various ways of forming the matrix m, one way is:

   ⍒⍤1∘.=⍨0 1 2 3
0 1 2 3
1 0 2 3
2 0 1 3
3 0 1 2

(Some authors waxed enthusiastic about this “magical matrix”.) In any case, a solution obtains readily from the above description: Form a matrix from the above individual planes; replace the 0 1 2 3 by ⍳⍵; and make an appropriate computation for the base case (when 0=⍵). See the 2015-07-12 entry below.

The Best perm Function

What is the “best” perm function I can write in APL? This “best” is a benchmark not only on my own understanding but also on advancements in APL over the years.

“Best” is a subjective and personal measure. Brevity comes into it but is not the only criteria. For example, {(∧/(⍳⍵)∊⍤1⊢t)⌿t←⍉(⍵⍴⍵)⊤⍳⍵*⍵} is the shortest known solution, but requires space and time exponential in the size of the result, and that disqualifies it from being “best”. The similarly inefficient {(∧/(⍳⍵)∊⍤1⊢t)⌿t←↑,⍳⍵⍴⍵} is shorter still, but does not work for 1=⍵.

• 1981, The N Queens Problem

    p←perm n;i;ind;t
   ⍝ all permutations of ⍳n
    p←(×n,n)⍴⎕io
    →(1≥n)⍴0
    t←perm n-1
    p←(0,n)⍴ind←⍳n
    i←n-~⎕io
   l10:p←(i,((i≠ind)/ind)[t]),[⎕io]p
    →(⎕io≤i←i-1)⍴l10

It was the fashion at the time that functions be written to work in either index-origin and therefore have ⎕io sprinkled hither, thither, and yon.

• 1987, Some Uses of { and }

   perm:  ⍪⌿k,⍤¯1 (⍙⍵-1){⍤¯ 1 k~⍤1 0 k←⍳⍵
       :  1≥⍵
       :  (1,⍵)⍴0

Written in Dictionary APL, wherein: ⍪⌿⍵ ←→ ⊃⍪⌿⊂⍤¯1⊢⍵ and differs from its definition in Dyalog APL; ⍙ is equivalent to ∇ in dfns; ⍺{⍵ ←→ (⊂⍺)⌷⍵; and ¯ by itself is infinity.

• 1990-2007

I worked on perm from time to time in this period, but in J rather than in APL. The results are described in a J essay and in a Vector article. The lessons translate directly into Dyalog APL.

• 2008, http://dfns.dyalog.com/n_pmat.htm

   pmat2←{{,[⍳2]↑(⊂⊂⎕io,1+⍵)⌷¨⍒¨↓∘.=⍨⍳1+1↓⍴⍵}⍣⍵⍉⍪⍬}

In retrospect, the power operator is not the best device to use, because the left operand function needs both the previous result (equivalent to perm ⍵-1) and ⍵. It is awkward to supply two arguments to that operand function, and the matter is finessed by computing the latter as 1+1↓⍴⍵.

In this formulation, ⍉⍪⍬ is rather circuitous compared to the equivalent 1 0⍴0. But the latter would have required a ⊢ or similar device to separate it from the right operand of the power operator.

• 2015-07-12

   perm←{0=⍵:1 0⍴0 ⋄ ,[⍳2](⊂0,1+∇ ¯1+⍵)⌷⍤1⍒⍤1∘.=⍨⍳⍵}

For a time I thought the base case can be ⍳1 0 instead of 1 0⍴0, and indeed the function works with that as the base case. Unfortunately (⍳1 0)≢1 0⍴0, having a different prototype and datatype.

• Future

Where might the improvements come from?

We are contemplating an under operator whose monadic case is f⍢g ⍵ ←→ g⍣¯1 f g ⍵. Therefore 1+∇ ¯1+⍵ ←→ ∇⍢(¯1∘+)⍵
Moreover, it is possible to define ≤⍵ as ⍵-1 (decrement) and ≥⍵ as ⍵+1 (increment), as in J; whence 1+∇ ¯1+⍵ ←→ ∇⍢≤⍵
Monadic = can be defined as in J, =⍵ ←→ (∪⍳⍨⍵)∘.=⍳≢⍵ (self-classify); whence ∘.=⍨⍳⍵ ←→ =⍳⍵

Putting it all together:

   perm←{0=⍵:1 0⍴0 ⋄ ,[⍳2](⊂0,∇⍢≤⍵)⌷⍤1⍒⍤1=⍳⍵}

We should do something about the ,[⍳2] :-)

M	T	W	T	F	S	S
« Nov
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

The tool of thought for

software solutions

Contact Us

The Importance of Communication, Transparency and Practice

Premium Capping Schemes: An important steering Mechanism in German Health Insurance

Parallel Programming with Futures/Isolates

Building a Web Application with MiServer

Postcard from Dyalog ’15 – Sicily

Permutations

Spoiler Alert

QUICK LINKS