Tag: programming

Book review: Exercises in Programming Style by Cristina Videira Lopes

exercises_in_programming_styleRecently our CIO has allowed us to claim one technical book per quarter on expenses as part of our continuing professional development. Needless to say since I was buying these books already I leapt at the opportunity! The first fruit of this policy is Exercises in Programming Style by Cristina Videira Lopes.

The book is modelled on Raymond Queneau’s book Exercises in Style which writes the same story in 99 different ways.

Exercises in Programming Style takes a simple exercise: counting the frequency of words in a file and reporting the top 25 words, and writes a program to do this in forty different styles spread across 10 sections.

The sections are historical, basic styles, function composition, objects and object interaction, reflection and metaprogramming, adversity, data-centric, concurrency, interactivity, and neural networks. The section on neural networks breaks the pattern with example programmes only handling small elements of the word frequency problem. The sections vary in size, the objects and object interaction is the largest.

Lopes talks about styles in terms of constraints, for example in the "Good old times" historical style there are no named variables and limited memory, in the "Letterbox" style objects pass messages to one another to prompt actions.

The shortest implementation of the example is in the "Code Golf" chapter with just six lines, other examples run to a couple of pages – a hundred lines or so. Lopes is somewhat opinionated as to style but quite balanced providing reasoning where unusual styles may be appropriate. This was most striking for me in the section on "Adversity" which discussed error-handling. Lopes suggests that a "Passive Aggressive" style with error handling all occurring at the top level in a try-except block is better than my error handling to date which has been more in the "Constructivist" (trapping errors but proceeding with defaults) or "Tantrum"(catching errors and refusing to proceed) style.

Sometimes the fit to the style format feels slightly forced, in particular in the chapters relating to neural networks but in the Data-Centric chapter I learnt how to implement spreadsheet-like functionality in Python which is interesting.

I’ve been programming for about 40 years but as a physical scientist analysing data or trying out numerical models rather than a professional developer. Exercises  brings together many bits and pieces of things I’ve learnt, often in the context of different languages. For a while I’ve had the feeling that I didn’t need to learn new languages, I needed to learn how to apply new techniques in my favoured language (Python) and this book does exactly that.

Once again I was bemused to see Python’s "gentleman’s agreement" methodology over certain matters. By convention methods of a class whose name start with an underscore are considered private but this isn’t enforced so if you really want to use a "private" method just go ahead. Similarly many object-oriented languages support a "this" keyword for the members of a class to refer to themselves. Python uses "self" but only by convention, you can specify "self" is "me" or whatever other name you please. The style format provides a nice way of demonstrating a feature of Python in a non-trivial but minimal functioning manner.

It is somewhat chastening to discover that many of the styles in this book had their abstract origins in the 1960s, shortly before I was born, entered experimental languages such as Smalltalk in the seventies where I would have read about them in computer magazines and became mainstream in the eighties and nineties in languages like C++, Java and Python, not long after the start of my programming career. Essentially, most of the action in this book has taken place during my lifetime! In physics we are used to the figures in our eponymous laws (Newton, Maxwell etc) being very long dead. In computing the same does not apply.

What I take away from Exercises is that to a fair degree modern programming languages can be used to implement a wide range of the ideas generated in computer science over the last 50 or so years so in improving your skill as a programmer learning new languages is not the highest priority. There is a benefit to learning new techniques in a language in which your are familiar. Clearly some languages are designed heavily to support a certain style, for example Haskell and functional programming but I found it easier to understand monads explained in the context of Python than in Haskell where everything was alien.

Exercises is surprisingly readable, the programs are well-documented and Lopes’ text is short but clear with references to further reading. It stands alongside Seven databases in Seven Weeks by Eric Redmond and Jim R. Wilson as a book that I will rave about and recommend to everyone!

Book Review: Scala for the Impatient by Cay S. Horstmann

scala_for_impatientI thought I should learn a new language, and Scala seemed like a good choice so I got Scala for the Impatient by Cay S. Horstmann.

Scala is a functional programming language which supports object orientation too. I’m attracted to it for a number of reasons. Firstly, I’m using or considering using a number of technologies which are based on Java – such as Elasticsearch, Neo4j and Spark. Although there are bindings to my favoured language, Python, for Spark in particular I feel a second class citizen. Scala, running as it does on the Java Virtual Machine, allows you to import Java functions easily and so gives better access to these systems.

I’m also attracted to Scala because it is rather less verbose than Java. It feels like some of the core aspects of the language ecosystem (like the dependency manager and testing frameworks) have matured rapidly although the range of available libraries is smaller than that of older languages.

Scala for the Impatient gets on with providing details of the language without much preamble. Its working assumption is that you’re somewhat familiar with Java and so concepts are explained relative to Java. I felt like it also made an assumption that you knew about the broad features of the language, since it made some use of forward referencing – where features are used in an example before being explained somewhat later in the book.

I must admit programming in Scala is a bit of a culture shock after Python. Partly because its compiled rather than interpreted, although the environment does what it can to elide this difference – Scala has an REPL (read-evaluate-print-loop) which works in the background by doing a quick compile. This allows you to play around with the language very easily. The second difference is static typing – Scala is a friendly statically typed language in the sense that if you initialise something with a string value then it doesn’t force you to tell it you want this to be a string. But everything does have a very definite type. It follows the modern hipster style of putting the type after the symbol name (i.e var somevariablename: Int = 5 ), as in Go rather than before, as in earlier languages (i.e int somevariablename = 5).

You have to declare new variables as either var or val. Variables (var) are mutable and values (val) are immutable. It strikes me that static typing and this feature should fix half of my programming errors which in a dynamically typed language are usually mis-spelling variable names, changing something as a side effect and putting the wrong type of thing into a variable – usually during I/O.

The book starts with chapters on basic control structures and data types, to classes and objects and collection data types. There are odd chapters on file handling and regular expressions, and also on XML processing which is built into the language, although it does not implement the popular xpath query language for XML. There is also a chapter on the parsing of formal grammars.

I found the chapter on futures and promises fascinating, these are relatively new ways to handle concurrency and parallelism which I hadn’t been exposed to before, I notice they have recently been introduced to Python.

Chapters on type parameters, advanced types and implicit types had me mostly confused although the early parts were straightforward enough. I’d heard of templating classes and data strctures but as someone programming mainly in a dynamically typed languages I hadn’t any call for them. I turns out templating is a whole lot more complicated than I realised!

My favourite chapter was the one on collections – perhaps because I’m a data scientists, and collections are where I put my data. Scala has a rich collection of collections and methods operating on collections. It avoids the abomination of the Python “dictionary” whose members are not ordered, as you might expect. Scala calls such a data structure a HashMap.

It remains to be seen whether reading, once again, chapters on object-oriented programming will result in me writing object-oriented programs. It hasn’t done in the past.

Scala for the Impatient doesn’t really cover the mechanics of installing Scala on your system or the development environment you might use but then such information tends to go stale fast and depends on platform. I will likely write a post on this, since installing Scala and its build tool, sbt, behind a corporate proxy was a small adventure.

Googling for further help I found myself at the Scala Cookbook by Alvin Alexander quite frequently. The definitive reference book for Scala is Programming in Scala by Martin Odersky, Lex Spoon and Bill Venners. Resorting to my now familiar technique of searching the acknowledgements for women working in the area, I found Susan Potter whose website is here.

Scala for the Impatient is well-named, it whistles through the language at a brisk pace, assuming you know how to program. It highlights the differences with Java, and provides you with the vocabulary to find out more.

Book review: BDD in Action by John Ferguson Smart

bddinactionBack to technical reading with this book BDD in Action by John Ferguson Smart. BDD stands for Behaviour Driven Development, a relatively new technique for specifying software requirements.

Behaviour Driven Development is an evolution of the Agile software development methodology which has project managers writing “stories” to describe features, and sees developers writing automated tests to guide the writing of code – this part is called “test driven development”. In behaviour driven development the project manager, along with their colleagues who may be business analysts, testers and developers, write structured, but still “natural language”, acceptance criteria which are translated into tests that are executed automatically.

Behaviour Driven Development was invented by Dan North whilst at Thoughtworks in London, there he wrote the first BDD test framework, JBehave and defined the language of the tests, called Gherkin. Gherkin looks like this:

Scenario: Register for online banking

Given that bill wants to register for online banking

When he submits his application online

Then his application should be created in a pending state

And he should be sent a PDF contract to sign by email

The scenario describes the feature that we are trying to implement, and the Given-When-Then steps describe the test, Given is the setup, When is an action and Then is the expected outcome. The developer writes so called “step definitions” which map to these steps and the BDD test framework arranges the running of the tests and the collection of results. There is a bit more to Gherkin than the snippet above encompasses, it can provide named variables and values, and even tables of values and outputs to be fed to the tests.

Subsequently BDD frameworks have been written for other languages, such as Lettuce for Python, SpecFlow for .NET and Cucumber for Ruby. There are higher level tools such as Thucydides and Cucumber Reports. These tools can be used to generate so-called “Living Documentation” where the documentation is guaranteed to describe the developed application because it describes the tests around which the application was built. Of course it is possible to write poorly considered tests and thus poor living documentation but the alternative is writing documentation completely divorced from code.

Reading the paragraph above I can see that for non-developers the choice of names may seem a bit whacky but that’s a foible of developers. I still have no idea how to pronounce Thucydides and my spelling of it is erratic.

BDD in Action describes all of this process including the non-technical parts of writing the test scenarios, and the execution of those scenarios using appropriate tools. It takes care to present examples across the range of languages and BDD frameworks. This is quite useful since it exposes some of how the different languages work and also shows the various dialects of Gherkin. BDD in Action also covers processes such as continuous integration and integration testing using Selenium.

As someone currently more on the developer side of the fence, rather than the (non-coding) project manager BDD seems to add additional layers of complexity since now I need a library to link my BDD style tests to actual code, and whilst I’m at it I may also include a test-runner library and a library for writing unit tests in BDD style (such as spock).

I’ve had some experience of managing Agile development and with that hat on BDD feels more promising, in principle I can now capture capabilities and feature requirements with my stakeholders in a language that my developers can run as code. Ideally BDD makes the project manager and stakeholders discuss the requirements in the form of explicit examples which the developers will code against.

BDD in Action has reminded why I haven’t spent much time using Java: everything is buried deep in directories, there are curly brackets everywhere and lots of boilerplate!

I suspect I won’t be using BDD in my current work but I’ll keep it in the back of my mind for when the need arises. Even without the tooling it is a different way of talking to stakeholders about requirements. From a technical point of view I’m thinking of switching my test naming conventions to methods like test_that_this_function_does_something arranged in classes named like WhenIWantToDoThisThing, as proposed in the text.

In keeping with my newfound sensitivity to the lack of women in technical writing, I scanned the acknowledgements for women and found Liz Keogh – who is also mentioned a number of times in the text as an experienced practioner of BDD. You can find Liz Keogh here. I did look for books on BDD written by women but I could find none.

If you want to know what Behaviour Driven Design is about, and you want to get a feel for how it looks technically in practice (without a firm commitment to any development language or libraries) then BDD in Action is a good place to start.

Book review: Working effectively with legacy code by Michael C. Feathers

legacy_codeWorking effectively with legacy code by Michael C. Feathers is one of the programmer’s classic texts. I’d seen it lying around the office at ScraperWiki but hadn’t picked it up since I didn’t think I was working with legacy code. I returned to read it having found it at the top of the list of recommended programming books from Stackoverflow at dev-books. Reading the description I learnt that it’s more a book about testing than about legacy code. Feathers defines legacy code simply as code without tests, he is of the Agile school of software development for whom tests are central.

With this in mind I thought it would be a useful read for me to improve my own code with the application of better tests and perhaps incidentally picking up some object-oriented style, in which I am currently lacking.

Following the theme of my previous blog post on women authors I note that there are two women authors in the 30 books on the dev-books list. It’s interesting that a number of books in the style of Working Effectively explicitly reference women as project managers, or testers in the text, i.e part of the team – I take this as a recognition that there exists a problem which needs to be addressed and this is pretty much the least you can do. However, beyond the family, friends and publishing team the acknowledgements mention one women in a lengthy list.

The book starts with a general overview of the techniques it will introduce, including the tools used to address them. These come down to testing frameworks and the refactoring tools found in many IDEs. The examples in the book are typically written in C++ or Java. I particularly liked the introduction of the ideas of the “seam”, a place where behaviour can be changed without editing the code and the “enabling point” – the place where a change can be made at that seam. A seam may be a class that can be replaced by another one, or a value altered. In desperate cases (in C) the preprocessor can be used to invoke test-time changes in the executed code.

There are then a set of chapters that answer questions that a legacy code-ridden developer might have such as:

  • I can’t get this class into a test harness
  • How do I know that I’m not breaking anything?
  • I need to make a change. What methods should I test?

This makes the book easy to navigate, if not a bit inelegant. It seems to me that the book addresses two problems in getting suitably sized pieces of code into a test harness. One of these is breaking the code into suitable sized pieces by, for example, extracting methods. The second is gaining independence of the pieces of code such that they can be tested without building a huge infrastructure up to support them.

Although I’ve not done any serious programming in Java or C++ I felt I generally understood the examples presented. My favoured language is Python, and the problems I tackle tend to be more amenable to a functional style of programming. Despite this I think many of the methods described are highly relevant – particularly those describing how to break down monster functions. The book is highly pragmatic, it accepts that the world is not full of applications in which beautiful structure diagrams are replicated by beautiful code.

There are differences between these compiled object-oriented languages and Python though. C#, Java, and C++ all have a collection of keywords (like public, private, protected, static and final) which control who can see what methods exist on a class and whether they can be over-ridden or replaced. These features present challenges for bringing legacy code under test. Python, on the other hand, has a “gentleman’s agreement” that method names starting with an underscore are private, but that’s it, and there are no mechanisms to prevent you using these “private” functions! Similarly, pretty much any method in Python can be over-ridden by monkey-patching. That’s to say if you don’t like a function in an imported library you can simply overwrite it with your own version after you’ve imported the library. This is not necessarily a good thing. A second difference is that Python comes with a unit testing framework and a mocking library rather than them being functionality which is third-party added. Although to be fair, the mocking library in Python was originally third party.

I’ve often felt I should programme in a more object-oriented style but this book has made me reconsider. It’s quite clear that spaghetti code can be written in an object oriented language as well as any other. And I suspect the data processing for which I am normally coding fits very well with a functional style of coding. The ideas of single responsibility functions, and testing still fit well with more functional programming styles.

Working effectively is readable and pragmatic. I suspect the developer’s dirty secret is that actually we wrote the legacy code that we’re now trying to fix.

Testing, testing…

testingtesting

This post was first published at ScraperWiki.

Data science is a distinct profession from software engineering. Data scientists may write a lot of computer code but the aim of their code is to answer questions about data. Sometimes they might want to expose the analysis software they have written to others in order they can answer questions for themselves, and this is where the pain starts. This is because writing code that only you will use and writing code someone else will use can be quite different.

ScraperWiki is a mixed environment, it contains people with a background in software engineering and those with a background in data analysis, like myself. Left to my own devices I will write code that simply does the analysis required. What it lacks is engineering. This might show up in its responses to the unexpected, its interactions with the user, its logical structure, or its reliability.

These shortcomings are addressed by good software engineering, an area of which I have theoretical knowledge but only sporadic implementation!

I was introduced to practical testing through pair programming: there were already tests in place for the code we were working on and we just ran them after each moderate chunk of code change. It was really easy. I was so excited by it that in the next session of pair programming, with someone else, it was me that suggested we added some tests!

My programming at ScraperWiki is typically in Python, for which there a number of useful testing tools. I typically work from Windows, using the Spyder IDE and I have a bash terminal window open to commit code to either BitBucket or Github. This second terminal turns out to be very handy for running tests.

Python has an internal testing mechanism called doctest which allows you to write tests into the top of a function in what looks like a comment. Typically these comprise a call to the function from a command prompt followed by the expected response. These tests are executed by running a command like:

 python -m doctest yourfile.py
def threshold_above(hist, threshold_value):
"""
>>> threshold_above(collections.Counter({518: 10, 520: 20, 530: 20, 525: 17}), 15)
[520, 530, 525]
"""
if not isinstance(hist,collections.Counter):
raise ValueError("requires collections.Counter")
above = [k for k, v in hist.items() if v > threshold_value]
return above

view raw
doctests.py
hosted with ❤ by GitHub

This is OK, and it’s “batteries included” but I find the mechanism a bit ugly. When you’re doing anything more complicated than testing inputs and outputs for individual functions, you want to use a more flexible mechanism like nose tools, with specloud to beautify the test output. The Git-Bash terminal on Windows needs a little shim in the form of ansicon to take full advantage of specloud’s features. Once you’re suitably tooled up, passed tests are marked with a vibrant, satisfying green and the failed tests by a dismal, uncomfortable red.

My latest project, a module which automatically extracts tables from PDF files, has testing. It divides into two categories: testing the overall functionality – handy as I fiddle with structure – and tests for mathematically or logically complex functions. In this second area I’ve started writing the tests before the functions, this is because often this type of function has a simple enough description and test case but implementation is a bit tricky. You can see the tests I have written for one of these functions here.

Testing isn’t as disruptive to my workflow as I thought it would be. Typically I would be repeatedly running my code as I explored my analysis making changes to a core pilot script. Using testing I can use multiple pilot scripts each testing different parts of my code; I’m testing more of my code more often and I can undertake moderate changes to my code, safe in the knowledge that my tests will limit the chances of unintended consequences.