It’s been another busy day at Dyalog ’14; there’s lots to write about and little time to do it!

Winners of the International APL Programming Contest 2014

This year, 710 registrations were received for the annual APL Programming Contest and the two main winners – Emil Bremer Orloff, student entrant from Aarhus Universitet, and Iryna Pashenkovska, professional entrant from SimCorp and a recent Engineering Mathematics graduate of National Taras Shevchenko University of Kyiv – won invitations to Dyalog ’14 as part of their prizes where they each made presentations about their contest submissions. Look forward to the videos of their presentations which will be available shortly!

Discussion Point: Data Binding – Reloaded

John Daintree showed more aspects of Dyalog’s data binding. Did you know that you can bind to XML stored in files? Did you know that you can create DataTables to bind to an APL matrix? The cddb database sample was re-worked to allow the GUI to bind to the data in a matrix as an alternative to vectors of namespace. Syncfusion controls were used to visually explore the cddb data.

John discussed changes that might come in 14.1, including better support for bound matrices.

Discussion Point: Syncfusion Controls

Late yesterday, Brian Becker showed how to take advantage of the Syncfusion Javascript controls that are bundled with Dyalog v14.0 on all platforms. This morning, Morten Kromberg and Michael Hughes followed up with demonstrations of how easy it is to enhance applications with Syncfusion controls for Windows Presentation Foundation, which are included with the Windows editions of version 14.0.

With Dyalog (as in any Microsoft.NET language), you have the choice of instantiating and manipulating WPF objects with APL language statements, or using declarative eXtensible Application Markup language, which can either be edited by UI design experts using WYSIWYG tools, or (of course) generated from APL.

Code sample for the RangeSlider:

     ∇ window←RangeSliderControl values;⎕USING;r;t
[1]   ⍝ Simple Sparkline example - inspired by Dick Bowman
[2]   ⍝ More at http://www.apldapldoo.info/apl/APL/Tutorials/WPF/SyncFusion/overview.html
[3]   ⍝ values: Numeric vector of values
[4]
[5]    :Section xaml
[6]        xaml←ScriptFollows ⍝ Picks up following comments as data
[7]    ⍝<Window
[8]    ⍝  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
[9]    ⍝  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
[10]   ⍝  xmlns:sftools="clr-namespace:Syncfusion.Windows.Tools.Controls;assembly=SyncFusion.Tools.WPF"
[11]   ⍝  Title="Syncfusion Range Slider" SizeToContent="Height" Width="400">
[12]   ⍝  <StackPanel>
[13]   ⍝    <sftools:RangeSliderControl Name="rangeslider" />
[14]   ⍝    <TextBox Name="textbox" MinHeight="25" MinWidth="100"
[15]   ⍝             HorizontalContentAlignment="Right"
[16]   ⍝             VerticalContentAlignment="Bottom"
[17]   ⍝             Background="Turquoise"/>
[18]   ⍝  </StackPanel>
[19]   ⍝</Window>
[20]   :EndSection
[21]
[22]   window←LoadXAML xaml
[23]   (r t)←window.FindName∘⊂¨'rangeslider' 'textbox'
[24]   ⎕USING←,⊂'Syncfusion.Windows.Tools.Controls,Syncfusion/4.5/Syncfusion.tools.WPF.dll'
[25]   r.onRangeChanged←'RangeChanged'
[26]   r.Range←⎕NEW DoubleRange values
[27]   window.Show
[28]
[29]   window.(slider textbox range)←r t r.Range ⍝ expose extra "properties"
     ∇

James Bond or Johnny English?

This year we enrolled our delegates in ‘Spy School’ and gave them the chance to be a spy for a few hours. In an interactive mission that included GPS tracking, teams were given a tablet preloaded with a series of spy related challenges that took place around the local area. The challenges were only activated when the GPS verified the team was in exactly the right location and points were awarded for completing each challenge successfully – the team with most points was the winner. To pile on the pressure, each team had an interactive map showing not only their location but also that of the other teams along with a live scoreboard.

The challenge certainly brought out the competitive side of the delegates and Eastbourne is probably still recovering from the shock of ten teams of APLers carrying out their covert missions around the town centre and promenade. When the final scores were in, it was a very closely run thing with Team Sprat in second place and Team Dover Sole, headed up by a super competitive John Daintree, as the eventual winners. At the final de-briefing we got to see the videos and photos submitted by each team which showed great creativity along with some very over active imaginations! Everyone had a fantastic afternoon with plenty of laughter and not too many injuries (although I am led to believe there may be a broken fingernail or two) and a big thank you to Pretty Clever Events for running the show.

Coming up…

Today is due to end with the banquet dinner in the Grand Hotel which we’ll try to include in tomorrow’s postcard.

We conclude events tomorrow with two Dyalog and two user presentations in the morning followed by the closing session at which all of Team Dyalog will be present and open to questions. In the afternoon are the remaining Technical Workshops.

This was originally posted to Catalyst PR’s Facebook page and is reproduced here to make it accessible to people who don’t have Facebook accounts.

For information on the presentations at Dyalog ’14, see http://www.dyalog.com/user-meetings/dyalog14.htm.

Maintainability

The first presentation I would like to highlight from today’s programme is by Gianfranco Alongi from Ericsson AB (Sweden). Those of you who read my posts from Dyalog ’13 last year will know that I am rather taken with Gianfranco’s approach to software development and, in particular, his insistence that it is a software programmer’s sole mission in life to leave a legacy (in the form of code) that is so simple and easy to understand that anyone who comes in later can pick up the ball and run with it, without having to re-invent the wheel and without risking destroying the application. Work smarter – not harder – is his mantra.

Gianfranco’s talk this year was called Let’s Talk about Maintainability!. The background for the talk is that any professional programmer obviously strives for maintainable code, well aware that an unmaintained codebase gradually transforms from a well-tended garden to a dark haunted forest that requires animal sacrifice if left to its own devices. Far too many systems have “that” piece of code – the unmaintained marsh, the mess that causes nausea and elevated blood pressure. Some people swear developers have lost parts of their souls in there and you can hear screaming if you work late enough for long enough…

During the talk Gianfranco explored some of the underlying mechanics of maintainability, presenting a definition of maintainability based on human cognition and sharing a language that might help when talking and reasoning about maintainability. Ultimately any programmer is responsible for maintaining the codebase; However, software entropy is real and we pay in blood, sweat and tears if we choose to ignore it. He insists – and quite rightly so – that maintainability is economics because the Cost of Change (CoC) goes up and up the longer a projects run and it’s measurable as it leads to smaller margins and ultimately smaller profit. Ultimately, you get to a point where it is no longer viable to throw more resources (more people or more overtime) into the mix, and you simply have to re-architect.

The psychology involved in software programming is fascinating! So is the shift from the days of programmers being proud of their ability to write code that other programmers may have difficulty understanding to a world where simplicity in the coding is the ultimate target – in order to address the increasing complexity in the applications.

Statistics

Stephen Mansour (PhD) has taught Statistics at the University of Scranton (U.S.A.) for the past 7 years. He is also working with The Carlisle Group – a company focused exclusively on creating software to support the fast growing mortgage and asset securitisation industry.

The Carlisle Group’s flagship product is CAS, a Collateral Analysis System highly specialised for global asset securitisation markets and used by participants in all facets of the mortgage business, including investment banks, commercial banks, accounting firms, bond insurers and loan originators.

Stephen’s presentation was named Taming Statistics with Limited-Domain Operators and took its offset in a new statistical package he has developed which is using Acre.

(Acre facilitates code management for APLers and was the subject of another talk today. Acre is available as open source software and can be downloaded for free from http://aplwiki.com/acre).

Asked why he felt there was a need for yet another stats package – after all, there are perfectly good packages available in the market, such as R, and even Excel has 87 stats functions, so what exactly is it that a stats package developed in APL can offer that the others can’t, the answer was: Defined Operators!

Stephen then demonstrated the new stats package by making some assumptions with regards to next year’s user meeting and the likelihood of the venue being over-subscribed (or not) based on a number of assumptions and relations.

The package does take advantage of the fact that R already connects into APL and Stephen ended the presentation by saying that the package is currently in beta. He is happy to share, so if you would like to review/try the new stats package please e-mail (Stephen.Mansour@scranton.edu) and specify whether you would like the R version or the native APL version.

The weather predictions have been spot on so far; it’s been another lovely sunny day at Eastbourne – here was the scene this morning at sunrise:

It was another busy day of presentations – read on for some highlights! After dinner tonight the evening will be rounded off with Morten and his Dancing ‘Bots – controlling two robots in parallel.

Discussion Point: Chart Design

As he demonstrated Chart Wizard – a front-end to help building SharpPlot charts – Nic pointed out a few useful tips about chart design:

• Transparency is your best friend
  SharpPlot supports transparency, which not only allows multiple layers of drawings to be readable, but also lightens the global visual impression, making charts more 21st-century
• 3D is not as cool as it looks
  The 2D alternatives (Bubble chart, Contour plot) are generally more readable, because perspective can hide data in automated 3D charts.
• Logarithmic scales are too often neglected
  They can improve readability in many cases, for example stock value, performance comparisons, or wealth repartition
• Box & Whiskers for categorised data exploration
  A very effective tool to explore the repartition of values amongst a un-modelled, yet categorised, numerical dataset.

Paul’s Poncho

The registration packs that delegates received included a stick of rock and a rain poncho generously donated by Optima Systems. As these items were unfamiliar to many from overseas, Paul Grosvenor – Optima Systems’ Managing Director – modelled them for us.

Behind The Scenes

Behind each presentation at Dyalog ’14 is a team of people taking care – hopefully mostly unnoticed – of all the AV requirements. Jason and Jonathan are working full-time in the AV suite and they are being assisted by a rota of Dyalog staff and Liam Flanagan, who has temporarily returned to us for the event, as microphone runners and camera operators. In total there are ten audio feeds and three video feeds, and every presentation is being recorded.

Generally, each presenter has different software and hardware needs and we allow everyone to use their own kit. In past years this has meant that changeovers have sometimes been less than seamless so this time we are experimenting with a system that allows one presenter’s equipment to be set up while the previous session is still taking place – this is giving a really noticeable speed-up in the changeover times.

Over the coming weeks the recordings will be edited and released online. The presentations will be released at the rate of approximately two per week as each generally takes around six to eight hours to edit and needs to be reviewed and approved by the presenter.

And Also …

Some of the many other things we saw and heard today:

• Kai Jaeger told us that software support such as Acre is essential to prevent chaos when there are multiple programmers working on the same project
• Gianfranco Alongi observed that maintenance is not cool, but said that if your product is not being maintained then the chances are it is already dead
• SimCorp maintains 1.7 million lines of APL code
• Stephen Mansour said that the biggest difference between Dyalog and other stats tools such as R is the existence of defined operators
• Nick did not use Powerpoint in his presentation but instead had a text-only equivalent that he wrote himself in coffeescript
• Aaron Hsu announced that his Co-dfns compiler would soon be available on GitHub for download
• Brian demonstrated MiServer 3’s ability to display CIA data
• Bjørn introduced the Dyalog Cryptographic Library which is now available for use

Did You Know?

John Scholes and Geoff Streeter started writing the Dyalog APL interpreter this day in 1981?

Tomorrow…

Tomorrow morning’s schedule includes two Dyalog and two user presentations. The afternoon will be dominated by the Viking Challenge (don’t forget to wear outdoor clothes and shoes that are suitable for walking!) followed by the prize ceremony for the International APL Programming Contest 2014. The two winners (one student and, for the first time, one non-student) will talk about how they achieved their victory. Everyone is encouraged to give this next generation of APLers their full support. The banquet will round off the day.

This was originally posted to Catalyst PR’s Facebook page and is reproduced here to make it accessible to people who don’t have Facebook accounts.

For information on the presentations at Dyalog ’14, see http://www.dyalog.com/user-meetings/dyalog14.htm.

After having spent yesterday in workshops with lots of hands-on examples, today was the kick off for the user meeting proper. The last time Dyalog held a global user meeting in the UK was in 2003, and 2014 has given us an all-time record attendance in terms of delegates.

DNA

My posts from Dyalog ’14 will focus on extracts from User Presentations, and the first presentation I would like to highlight is There’s DNA Everywhere – an Opportunity for APL by Charles Brenner PhD (http://dna-view.com).

Charles describes himself as consulting in forensic mathematics and his DNA-View software programme is used by 100 laboratories in every continent (except Antarctica) for academic activities in mathematics, biostatistics and various aspects of population genetics.

Charles used the example of gun crime to illustrate: “Use a gun in a crime and, thanks to recent biochemical advances, you’ll likely leave enough DNA to be detected. However, often the gun has also been handled by others and, consequently, the analysis of DNA mixtures has become increasingly important. The market for good DNA mixture software, therefore, has led me to recent and rapid progress in my long-delayed venture to convert my forensic DNA interpretation work into the modern world of Dyalog. Many people enjoy the scientific aspects of the story and I enjoy relating how, in my view, this project exemplifies the thesis that APL is a tool of thought. I credit APL with leading me to elegant and, therefore, very fast and flexible solutions to a problem for which the competing solutions are lumbered by complicated statistical and Monte Carlo methods.”

Elegance and simplicity lead to several concrete benefits. The first is conceptual development.

“I made notes in APL as solutions to various aspects of the problem occurred to me last year. These brief APL notes ensured that the ideas really made sense, that they worked together and that I would not forget them. The brevity also revealed a simple but important point that others had overlooked: nesting the computation loops in the right order saves orders of magnitude in computation time. For another, execution speed makes it much easier to see the forest in many ways – testing, developing, designing. Competing programs take hours to find a partial answer, then having worked that hard succumb to a natural tendency to call it a day. The APL “DNA-VIEW Mixture Solution” program does the same in seconds, which makes it much easier to think through to the fact that there’s a lot more work to do before the result is logically defensible.” Charles explained.

At the end of 2013 Charles implemented the Mixture Solution and tested it on examples including a set of five proficiency exercises created at the National Institute of Standards and Technology. More than one hundred entrants, including the leading competition, had contributed analyses. One of the competitors (supported by years and millions of dollars of government grants) got four problems right and came close on the fifth, viewing it as a three person mixture although in fact it was four. Mixture Solution alone correctly analysed all five – as a bonus it correctly diagnosed one suspect as a mixed race person.

“We APLers try to be modest but it’s not always easy. Sometimes there’s just not much to be modest about!” Charles stated.

Having now reached a point where he has developed a superior, fast, elegant software solution to a massively complex DNA Mixture analysis problem, Charles ended the presentation with a plea for help. First and foremost he is looking for someone who can help carry on with the further software development using Dyalog. This could be a fantastic opportunity for someone with an interest in forensics, maths, computer science and software programming in Dyalog – “living in the Bay area”.

ELSI

The Finnish Pension System is currently undergoing a reform and there is a strong need to microsimulate various parameters to help make decisions regarding future pension schemes.

Heikki Tikanmäki from the Finnish Centre for Pensions gave us a presentation of ELSI – the Pension microsimulation model that has been built in the Finnish Centre for Pensions using Dyalog.

ELSI models the statutory pension system of Finland. It is used for reform analysis and long-term projections. In a typical simulation run, they model 500,000 individuals over 70 years.

By using Dyalog not only do they retain compatibility with older macromodels, population and employment projections, they also gain flexibility for modelling various proposals for a new pension scheme.

The model is built in modules with each module running as independently as possible. All the simulation data are held in component files. Depending on the complexity of each module, it takes 10-40 minutes to run using version 12.1 – without doing any performance optimisation.

Heikki told us that by upgrading to Dyalog version 14.0 the model runs 20% faster and the simulation data are now only taking up 16% space in comparison to the 5 GB size they previously had.

Future plans include implementing the use of parallel processors for the computational tasks in ELSI, and at that point in time he expects at least a further 50% reduction in the time it takes to run the entire microsimulation model.

Monday opened with registration for what is the biggest Dyalog user meeting on record – it could have been even bigger but we had to turn some people away because there was no more room available. In total there are 126 attendees (even more that when we went to print with the programme!) made up from 95 delegates, 12 spouses and partners, and 19 Dyalog employees from around the world as shown.

Discussion Point: DNA Analysis

Charles Brenner performs forensic analysis of DNA and DNA mixtures – an intensive mathematical process. Others use complicated statistical and Monte Carlo methods but by using APL as his tool of thought, Charles has devised techniques which are more accurate and many times faster than competing applications supported by multi-million dollar funding.

Discussion Point: A Different Kind of Selfie

In today’s session “The Tuning Pipeline”, Roger Hui noted that for the index-of family of functions, a faster algorithm is often possible if the left and right arguments are “the same”, up to a factor of two. “The same” means the that the values have identical references so that they can be compared with a simple and quick check. For example, x⍳x is a selfie but x⍳0+x or x⍳(⍴x)⍴x are not.

For example, for the inverted-table index-of 8⌶,

      u←(' ',⎕a,⎕d)[?8e4 30⍴37]
      x←u[?1e5⍴≢u;]
      p←,⊂x
      q←,⊂x
      cmpx 'p(8⌶)p' 'p(8⌶)q'
  p(8⌶)p → 3.98E¯3 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕                      
  p(8⌶)q → 8.72E¯3 | +118% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

Selfies occur in less obvious places – when finding uniques (∪x), in the key operator, and in x∧.=⍉y as well as in x⍳x and ⍳⍨x. You can try this yourself for various datatypes. A selfie which is not faster is an opportunity for a performance improvement.

Discussion Point: Sorting is Faster Than Grading

Also in “The Tuning Pipeline” session, Roger Hui and Kimmo Kekäläinen looked at some of the recent performance improvements in Dyalog. It has long had idioms for sorting ({⍵[⍋⍵]}, {⍵[⍋⍵;]}, etc.) Interestingly, for some common datatypes, sorting is faster than grading. You can try this yourself for various datatypes:

      cmpx '{⍵[⍋⍵]}x' '⍋x' ⊣ x←?1e6⍴2e9
  {⍵[⍋⍵]}x → 3.23E¯2 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕                         
* ⍋x       → 8.34E¯2 | +158% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

The point is written up in this essay on the J website. The webpage uses J but the explanation applies to APL (or any other language). Quoting from the webpage: “Grade needs to keep track of the argument array and the list of indices. Sort just needs to keep track of the first. … When the argument items can be manipulated efficiently, as would be the case when the items are machine units (1, 2, 4, or 8 bytes), then sort can eschew a separate index array.”

John Scholes’ “Distractions” – An Objective Review

An awed silence gripped the room as John unveiled ground-breaking techniques for minimising life’s distractions and maximising programming productivity. It is too early to say how far-reaching this approach will turn out be in real life situations and whether it will affect The Global Economy as a whole. John appeared to have all categories of distraction covered. Let’s see. Like Woodstock ’69 and The Isle of Wight ’70, in years to come you may be able to say “I was there”.

And Also …

Some of the many other things we saw and heard today:

• Dyalog has taken on two new employees since Dyalog ’13
• John Daintree can’t take selfies but embraces high resolution touch devices
• Data files used in Finnish pension microsimulations reduced to one sixth of their size when component file compression was enabled
• MyDyalog launched live at the user meeting
• Fiona wants to promote your APL application in a banner on the Dyalog homepage
• Morten computed Mandlebrot set images, performing the calculation in parallel with isolates running on servers in Holland and Hong Kong

Tomorrow…

Tomorrow’s schedule features five user presentations and four Dyalog presentations covering the themes of code management and reuse, performance, presentation tools and cryptography. In the evening Morten will demonstrate the progress he has made with the ‘bots (something that should be very familiar to regular readers of the Dyalog blog!).

Today at Dyalog ’14 was a day of Technical Workshops: six half-day sessions and one all-day session were held, and there will be more at the end of the user meeting on Thursday.

This evening Gitte will informally welcome the attendees before Vibeke introduces the scavenger hunt and Karen and Jason run a traditional English pub quiz so that delegates can get to know each other and the host country a bit more.

Getting Involved

This year we plan to keep you more up to date than ever before with what is happening at the user meeting. Programme updates and supporting materials will be made available on our website. There will be blog posts each day and we are using the Twitter hashtag #dyalog14 on our tweets – we encourage everyone to join in with us whether in Eastbourne or not.

Preparations

For Team Dyalog, the start of the user meeting is the culmination of preparations spanning weeks or even months. There have been some obstacles along the way – not least the fire on the pier, which meant a replacement banquet dinner venue had to be hastily found – but the presentations are ready and everything kicks off in the morning. We are looking forward to a busy but enjoyable week!

* See this blog post

Tomorrow…

Dyalog ’14 formally begins tomorrow. We expect a busy day commencing with registration and followed by ten presentations in total, starting with a formal welcome from Gitte. As has become the custom, there will be a mixture of presentations from Dyalog Ltd, guests and invited speakers throughout the week. Monday traditionally features Dyalog presentations and this year is no exception – but we will also have Charles Bremner and Heikki Tikanmäki talking about forensic DNA analysis and pension modelling respectively. Two years ago, by popular request, we abandoned the practice of running presentations in parallel (extending the programme by a day to allow this) and all of the presentations will run sequentially again this year.

There are planned evening activities for each of the conference nights. After dinner tomorrow, John Scholes will present his observations on the subject of “distractions” – those who have attended his evening presentations before will know they can be highly entertaining!