Response to Feedback on Cut, Under and Merge

At Dyalog ’15 John Scholes and I presented proposals for future operators cut, under, and merge. Following the release of the video of this presentation, we received some feedback from a user. Our response to the feedback may be of wider interest.

  1. It’s early days yet for cut, under, and merge (they are tentatively planned for release in version 16.0), and one of the reasons for that is so that we can receive feedback from the user community. So thank you for your comments.
  2. We take our inspiration from many sources, and while it’s true that J has cut, under, and merge, cut saw the light of day in Rationalized APL (1983), under in Operators and Functions (1978), and merge in Rationalized APL (1983) again. And even before that, the ideas have been in mathematics or functional programming for many years.Wherever the inspiration came from, we don’t take them just on faith, just because they are in J.Even now, even in these early days, cut, under, and merge have undergone a vigorous internal debate. The discussion is informed by the experience in other contexts, J or otherwise.
  3. We are interested in what you think are the “few fundamental ideas expressed by a handful of mathematical symbols”, what you say is the definition of APL. From our perspective we have not violated and will not violate what we think are the fundamental ideas. But perhaps our set of fundamental ideas differ from yours?
  4. No one here is thinking of adopting the J spelling scheme (with the dots and colons).
  5. The Dyalog development team are in unanimous agreement that dfns, ⍺⍵-functions, are A Good Thing. As we see it, the recent and proposed additions to Dyalog APL have only enhanced the elegance and utility of dfns. For example, {(+⌿⍵)÷≢⍵}⍤r to compute the mean of not just vectors but subarrays of a specified rank, {⍺⍵}⌸ to compute the unique keys and the corresponding indices, {⍺(+⌿⍵)}⌸ to compute the unique keys and corresponding sums, and {⊂Cut ¯2⊢⍵,','}Cut ¯2 ⊢x~⎕av[4] to partition CSV x into its columns and cells.
  6. Cut, under, and merge are described and documented in discussion papers accompanying the video. If you have not already read them, they can be found in the Dyalog ’15 webpage and specifically here. Please tell us what more can be done as far as documentation, and what parts specifically are unclear.
  7. We have previously (and recently) examined the question of adding quaternions to Dyalog. Currently there are no plans to add them.
  8. Finally, examples of under from everyday life. Of course the processes involved can be composed, so that for example “dinner” can be:
    open the fridge
        take the food
    close the fridge

    but it can also be a more elaborate:

    take plate from cupboard
        open the fridge
            take the food
        close the fridge
     
        put food on plate
            open mouth
                put food in mouth
            close mouth
        clean plate
    put plate back into cupboard

    As remarked in the Dyalog ’15 presentation, once you are attuned to it you can see under everywhere, sometimes in subtle ways: Ashes to ashes, dust to dust.

Dyalog ’15 – An Unexpected Viking Challenge

etnaAfter many months of planning and preparation it was finally time to head from the UK to Sicily for Dyalog ’15. Sun, sea and an excellent conference facility were awaiting us at the Atahotel Naxos Beach in Giardini Naxos and this year, because it would be very hot, we had decided to forgo the traditional Viking Challenge team activity for a more relaxed excursion by coach to see the craters at the top of Mount Etna – Europe’s largest active volcano.

water_everywhereDyalog ’15 kicked off on Sunday morning and by Tuesday everything was going to plan and we were stubbornly ignoring the forecast for storms and rain on Wednesday – after all, the forecasters always over-exaggerate don’t they?
In this case it seems not as we woke up Wednesday morning to torrential rain and had to wade from our rooms (sometime through water up to our knees) to the conference centre.

flood_collageThere we were greeted by more floods and rain only this time it was inside the main conference room and perilously close to the desk with all our AV equipment on it. Time for an emergency change of room but where do you fit 80+ people at short notice? A room was found, the projectors and microphones were set up ready for the start of the day – and we were only 15 minutes behind schedule! OK, so maybe it was a tight squeeze but by then everyone was bonding over flood stories and determined not to beaten by a “bit of weather”.

banquetJust when we thought it couldn’t get worse, we then found out that the beautiful seaside terrace restaurant, where the Dyalog ’15 banquet was due to take place that evening, was also flooded so Banquet Plan C (Banquet Plan B was also under water) was implemented and the whole event moved indoors – not such a stunning location but at least we would all be dry.

During the morning the rain eased off, the floods subsided and our planned excursion to Etna during the afternoon was looking more and more likely. We finally got the go ahead and 69 brave souls set off after lunch to battle with the elements, optimistically hoping the weather would improve as the afternoon went on. However, once we reached 1900m we were greeted by yet more torrential rain and a pretty bad thunderstorm. Were we deterred? Oh no, we just headed off into the storm to stomp around the Silvestri Craters for a bit – although one delegate did, rather hysterically, point out that it probably wasn’t very sensible to be waving an umbrella around in a thunderstorm at that height. Eventually a very wet, cold and bedraggled bunch got back on the coaches but we all survived the adventure, made it back to the hotel and most agreed we had had a lot of fun!
etna_collage
And the moral of the story? It was beautifully summed up by Gitte Christensen in her closing speech: “If you don’t have a Viking Challenge at a Dyalog User Meeting then one will be provided for you!”

Tuesday at Dyalog ’15

IMG_5010MiServer UI Controls

Dan Baronet and Adám Brudzewsky continued yesterday’s presentation by Morten Kromberg on Cross-platform User Interfaces by demonstrating a lot of varied interface elements that are available when creating front-ends in MiServer. Especially encouraging were the special controls designed by Dyalog. These make it possible to have arrays render as complete super-elements (e.g. a table). No need to worry about the many underlying HTML elements making up the super-structure (e.g. table rows and their table cells).

IMG_4970The Dyalog Project Project

Morten Kromberg outlined his plans for a brand new tool to ease development and getting started with APL: An online repository for APL code – software packages that may be retrieved and installed on a computer. This is a significant step in ensuring the future, as a newcomer to programming (if ignoring all the other advantages to APL) often looks at amount of easily available pre-existing code when choosing a language.

IMG_5123Data Binding – The Matrix / Waste Time with John Daintree

Light-hearted as always, John Daintree introduced us all to Dyalog’s already implemented capabilities in making user interfaces automatically update when values the datatable are changed. Obviously this is best described in a video, which will be available soon on video.dyalog.com. Meanwhile, there is an excellent demo on APL Wiki. After John’s presentation, a user demonstrated how he with minimal time effort set up a modern-looking dashboard for presenting numerous varied data.

External Workspaces

Morten Kromberg introduced a new facility by which you can save your code and access it as a memory mapped file which can be shared by the users. This is an advantage when you have to watch how much of the workspace is consumed by the code-base. Stig Nielsen from SimCorp told us how they have applied these external workspaces with much success. Not only can many users access the same workspace, but even users on a Citrix farm can access the file simultaneously. Using this facility, enables SimCorp to start using futures and isolates that would otherwise exhaust the memory available on their costumers’ server farms.

TamStat, a Statistical Package

Blog post from presentation by Stephen M Mansour PhD, The Carlisle Group as well as the University of Scranton,  and Jerry Brennan PhD, A former teacher of Statistics in Hawaii, who has now moved being a fulltime Psycologist.

By Vibeke Ulmann #Dyalog15

Last year I managed to squeeze in a mention of TamStat in a blog post. This year we will do a bit more on this new application which has been developed by Stephen Mansour. The name TamStat stems from ‘Taming Statistics’ and Stephen willingly admitted that he stole the paradigm from Ken Iversson who once talked about ‘Taming Maths’.

Stephen teaches Statistics at the University of Scranton – outside as working at the Carlisle Group. The basis for the development of TamStat was that he wanted to create a one or two semester stats course at the University. In order to create what he considers to be an easy to use, APL based statistics programme, he has included a seamless interface to R.

TamStats functionality can be divided into four different groups:
Summary statistics
Probability Distributions – based on theoretical models,
Relations and
Logic

The underlying implementation is a set of operators representing the different distributions with operands specifying the conditions and the Relations, or Logical conditions which apply

As an example Stephen then conducted a comparison of a Movie Theatre with 130 seats, with an average of 100 attendees  and a 20 seat deviation. He ran the comparison in Excel, Minitab, R and TamStat to show the audience how easy it was to create a close to normal English language expression in TamStat to get the statistical result.

Since last year Stephen has been adding a lot of new features and not least documentation. There is now a 60 page user guide both in print and as a PDF download, a Reference Card  in fold out hardcopy, based on the same structure as the Dyalog APL reference card. The reference card is also available as a download. There is an Excel import functions, and operational R interface. It is now possible to work with Bayesian statistics – a request from last year.

TamStat is now available in 3 versions (almost 4) namely

  • A Dyalog workspace for APL users
  • A downloadable standalone App for students and non-APL users (with ASCII input).No set-up required just get it and start.
  • A Web application  – similar to TryAPL – using MiServer for casual users
  • A Computer programme for professional users (currently under development in Carlisle Group)

Jerry Brennan then took over demonstrating his JMB.APLcloud.com, which is an educational site teaching APL. Implemented in APL of course and using MiServer as the vehicle to bring the content to the web.

Jerry is currently running on MiServer 2.1 and he is in the process of moving to MiServer 3.0.

As Stephen, Jerry has made available documentation, and examples, in the form of games and other ‘fun’ examples to attract the attention of a young audience. He has included drag and drop facilities, to make it even easier for people who have no prior knowledge of Dyalog APL to get going.

Jerry has included TamStat on his website including all the examples from Stephen’s User Guide allowing the users to interactively experiment with the statistics.

Outside of this, he is working with ‘Hacker Highschool’ who work with security awareness for teens and have 1.4 Million subscribers. Hacker Highschool takes an extremely ethical approach, so he is not concerned about making the application available to students.

Stephen Mansour would like to run a workshop on his “Statistical Language” at the Dyalog User meeting next year if there is sufficient interest, so asked delegates and blog readers to let Dyalog know if we should include this in the next User Meeting.

Using Co-dfns to Accelerate APL Code

Blog post from Aaron Hsu’s (Indiana University, USA) presentation Wednesday 9th September.

By Vibeke Ulmann #Dyalog15

For the past 2-3 years Aaron and Dyalog have been collaborating on creating a new compiler. This is an Open Source project and the compiler – which is currently in ‘Version 0.5’, is available as a zip download from Github. Aaron started by showing us how to download, unzip and set-up the compiler. It is available for both Windows and Linux platforms.

This was a live coding session (working on the Window platform) where Aaron demonstrate some of the feature/functionality expected to go into Version 1 of the compiler. He then proceeded to do a number of blackscholes benchmark, working various Co-dfns implementations, in order to demonstrate the order of magnitude of speedup’s for chosen expressions. The performance speedup’s achieved were  60% -93% or, as Aaron prefers to see it, 3 – 10 times speed-,up , so significant improvements can be had by using the compiler. However, there is still work to be done in terms of implementing more expressions, and Aaron encouraged everyone to download the compiler, play with it, and most importantly – try to break it – as he needs feedback in order to start bug fixing.

The compiler is targeted solely at dfns and you will need access to a c-compiler on your machine. Also only 0-origin code is accepted for now.

There is a lot of additional information available on the Github wiki, so please visit Github, get hold of the Compiler and start playing!!

How I won the Dyalog Programming Contest

Blog post from Arianna Locatelli’s presentation on how she won the Dyalog APL programming contest 2015.

By Vibeke Ulmann #Dyalog15

Introduction

Now it it’s 7th year, the annual Dyalog APL Programming Contest continues to attract a lot of interest. One of our favourite sessions at the Annual User Meeting, is when we get the opportunity to meet the winners, and hear how they won the Grand Prize.

Again this year we collaborated with SQORE/Studentcompetitions, and the contest had 2 phases.

  • Phase 1 had 10 ‘simple’ one liner style problems
  • Phase 2 had low, medium and difficult problems.

This year we had 1786 registrants, 42 submissions for Phase 1 and 16 submissions for Phase 2. We had participants from all over the globe – perhaps with the exception of Antartica.

A deep felt THANK YOU goes to our sponsors: APL Italiana, Dyalog APL, Fiserv, and SimCorp, as well as a number of sponsors who prefer to remain anonymous.

The Winner

Arianna Locatelli is 17 years old and is currently a fourth year student in Liceo Classico S. M. Legnani, in Saronno, a little town near Milan. She was one of the students who took part in Roberto Minervini’s Dyalog APL class at her school. Reference Blog post  How to Save an Italian School with APL.

Competition winner gets certificate

Arianna was presented with her winner Certificate by Walt Syzonenko from Fiserv

Despite the fact that Arianna’s main interest is in humanistic subjects, especially Latin and ancient Greek, and despite the fact that informatics is not taught in her school, she took at keen interest in learning the Dyalog APL language – so much so – that when her teacher Roberto Minervini encouraged her to sign up for the 7th Programming Contest,  she took on the challenge.

“As a high School student I thought I had little chance of winning. I thought there would be an unbreachable gap between the humanistic subjects I favour, and Dyalog APL. But I discovered that abtract thinking and understanding, as well as discussing some of the problems in the contest with my fellow students made this a very interesting experience.I also found Bernard Legrands book Mastering Dyalog APL (http://www.dyalog.com/mastering-dyalog-apl.htm), as well as using the Internet very useful.” Arianne said when she started her presentation.

She then processed to give a us  a truly fluent presentation, first presenting the logical solution to a problem then the APL implementation. It was particularly impressive to see how her thought processes led her to master some of the tasks with the highest degree of difficulty and complexity in Phase 2.

Asked about her favourite APL expression – without hesitation she said:  *Find*.

We all congratulate Arianne with her winning submission – which is particularly impressive considering her young age.  We hope that although it is her intention to study humanistic subjects, she will be able to find topics which will encourage her to continue to use the Dyalog APL language.