The final marathon

Right now work on CMS is tough. We have tight deadlines to meet and a lot of work to do. This is what happens every time we get new data, and despite the increased workload it is my favourite time to be a physicist. What usually happens is that we target the Moriond conferences in March, but this time there is a December Jamboree for rapid analyses, and I'm working on one of them. The Jamboree and Moriond are my final push to get some physics results out, physics results that I'll be proud of and will make me nostalgic years from now. This is what I've been waiting for the past two years, building up my tools slowly and getting them ready, in anticipation of the data that turns our world upside down. There was some initial excitement over a single very high mass event, which was completely unexpected. Since then we've been keeping an extra look out for high mass events in the data set, and this has only made the curent work more exciting and competitive.

Unfortunately this level of work is not sustainable, even without any other commitments. As one of my friends pointed out in a recent (and probably final) visit to CERN, it's not unusual to do 40 hours of work in two days when things get tough. On top this, there is travel, teaching, and other meetings that take a long time. When I moved to CERN, I moved there to work, and to immerse myself in the life of the lab, which meant more work. It was very exciting and rewarding, but also exhausting. Since moving to Brussels I've been told to restrain my enthusiasm and to not lose myself in work, which isn't why I came here. I came to work hard, to give physics one last chance to show me that I still love it. Looking to the future, I realised that I don't want to keep following the same patterns. Whatever I work at, I'll always work hard and have long hours, but for the next few years I want to be able to set my own deadlines and any rush will be self inflicted.

A quick trip to Venice. It was pleasant, but did rather distract from the main event.

So I find myself slogging through the final marathon, a final blaze of glory before it all ends, and for now, I love it. If I had my own way I'd do nothing else, and for a few weeks I'd immserse myself in the work, going from one cross check to another and measuring all the spectra I could get manage. Knowing that this is the last time makes it all the more enjoyable, and more frustrating when other commitments keep me away from this work. These few months are the whole reason I came, and the reason I waited two years for the data. This is what make physics fun, this is what makes physics worth the time and effort.

Moonlighting

For the past few months I've been rather busy (hence the lack of updates) with some outreach activities. For a long time I've wanted to create a lightweight particle physics experiment simulator that can run in the browser. An offshoot of this has been a trigger simulator and this has been used in various physics shows across the UK. These shows have developed from the standup comedy I performed last year, into a series of interactive comedic lectures where I write apps that allow the audience to interact. As with any project, it has its fair share of frustrations, but overall I'm very pleased and excited by how it's progressing. The show itself is roughly split in two halves, with the first half discussing cognitive biases, and the second half taking the audience through the scientific method, using the discovery of the Higgs boson as an example. They use the apps to collect and analyse data, and we end the show with the discovery of the boson.

It ticks a lot of the boxes that I was looking for in a project:

• It allows me to develop some JavaScript skills in a new environment.
• It forces me to write for portable devices.
• It involves writing a user interface and testing it on people.
• It develops my communication skills.
• There's opportunity for travel in the UK
• It's slowly building up a range of useful contacts for future work.
• It involves collaboration with colleagues in the UK.
• There are social media elements that I need to build up.

The first draft of the app in action.

To be honest it is quite a workload to commit to, and there are still some elements that have to come together. However the timeline is long, the progress has been quite rapid, and the audiences seem to love the show (and give useful feedback at the end.) As with most of my projects, this is a stepping stone to something bigger and better. Interacting with the public (both the audience and people testing my apps) exposes me to new skills I haven't need to explore much before, and each iteration improves my skills at making the interaction intuitive. For example, no matter how rough a draft is, and no matter how much I tell someone about a project, they still assume that if an app doesn't acknowledge an input then it hasn't seen it. That means for every interaction I need to add something to let the user know things are happening, even in the roughest, most basic draft. Suddenly even trivial changes take a while to implement.

There is scope to extend the project beyond January, and for now I'm ambivalent about doing so. I already have many plans for what to do in the following year and I don't want to overburden myself with free projects. There's also a constant tension between the education and the entertainment, and I want to focus more on the education. Unfortunately that doesn't sell many tickets, and it's harder to deliver. At the same time the most lucrative elements of the show are not so much focused on the science itself, but more about cognitive biases. That's a fun and exciting avenue to pursue, bue it may require the help of someone with a degree in psychology to back up some of the claims we make.

During the show itself.

This project is a nice counter balance to my research, with a focus on communication rather than analysis, ease of use rather than problem solving, and rediscovery rather than research. Even the medium of the browser is different, more direct and fast paced (and in my opinion, more pleasant to work with) than the data analysis on the GRID. That said, the standards are higher. The public expect things to work flawlessly first time, and do not tolerate quick fixes or works in progress.

Much of the work is already done, but there is still a lot to finish up, and with the data analysis in full swing I don't have much time to dedicate to it these days. Right now I'm hurtling across the UK on a train to give a seminar in Liverpool, while my analysis jobs are being resubmitted. There's still a lot of physics to be done between now and the end of my contract and I have to make time for both my job and my outreach. I'm never bored with this much physics!

Embracing uncertainty

I've spent the past three weeks at CERN for shifts (something I'll blog about another time) and bumped into many old friends and colleagues. Some of them have asked me about what I've got planned for the next year or so. That's not an easy question to answer, and not one that I want to answer right now. My life from one week to the next is very diverse, which means I have problems predicting what I'll be doing a week from now, let alone a year from now. That's how I like it. I love it that my life changes giving me new experiences all the time. On top of that, if life has taught me anything in the past decade it's that it's okay to not have a firm plan.

I'm not sure why other people seem to find uncertainty uncomfortable. For me it means freedom. It means not tying myself down and being predicatble or boring or bored. I'm currently in a physics lab where we have to routinely deal with lack of knowledge and where we have to find the answers ourselves, if we're lucky enough that we can obtain some answers. We all know what the job market is like in the field and that it's normal to move around, and that the number of opportunities outside the field is vast compared to the number inside. Yet somehow people find my lack of a job and a place to live in six months disturbing. That's how I found myself two years ago, with my SMU contract winding up and searching for a new job. Within a few months of looking I found a great job in Brussels in a new country in a city I'd never lived in before. I went from no planned job and no planned abode to finding a decent position and a flat in a foreign country. It was all timed so that one contract ended the day before the next one began. That was all down to embracing the uncertainty and taking advantage of the opportunities of the time.

Uncertainty is when you meet a colleague of a colleague in Paris and he asks you to join him at the South Pole for some world class science.

I don't know why people ask me what my plans are. Is to make me feel better, or them? Is it reassure them that the world isn't a scary place? Sometimes it feels invasive. These uncertainties are my own, not theirs, and they're not entitled to an answer to these tough questions, even if I had them. I suppose in some ways it's easier for me as someone single with no dependents. I can't maintain a stable relationahip with this kind of lifestyle, and pets and children are certainly out of the question.

I won't pretend that it's always easy. Occassionally I am afraid of the future and the uncertainty that comes with it. Sometimes I think I must be mad to leave the comfortable bubble of academia. Then I look back at the other times in my life with hard decisions, the times I moved everything I had into a new, foreign place, and was a better person for it. One of the reasons I moved to Brussels was exactly because it was a new place and living in a foreign country is good for personal development. I've done that three times now, and aside from learning French again I don't think there's much else to be gained from doing it again. The kind of personal growth I need now involves creating and leading my own projects. There's a huge scope for learning there and so it brings its own steep learning curve and uncertainty. I don't know where I'll be what I'll be doing a year from now because I don't know much about the steps I'm going to take in the next few months. I can say exactly the same about every year since I finished my undergraduate course, and those years have given me more than I ever would have asked for.

When work and pastimes collide

Someone asked me to write a few paragraphs about what I like about my work and coding in general, so here's what I sent. It quite nicely summarises the transition from physics based problem solving, and other, further reaching topics.

I'm Aidan and I use computers all the time for work and as a hobby! I work on the CMS experiment at CERN and it's my job to analyse the data that we collect. It's a very exciting job because we're using some of the world's largest data sets and the world's most powerful computing systems, and we're the first people to see the results. It's not a simple job, because the task is so large, and there are many opportunities for mistakes and failures, of both humans and computers. The final results we show the public are usually just a graph, or s single number, or even a diagram of a single physics event, but there are hundreds of tiny steps between getting the data and showing those results. That's what I do, and there's something very satisfying about making that possible.

Some of the computing resources available at CERN. (CERN Computing)

It's all about problem solving and being creative in how to approach these problems. At CERN we have access to so many different resources, and we are given the freedom to use them as we see fit. We're faced with some of the most challenging and promising data in the world, we're given some of the most impressive tools to analyse them, and we can play about with whatever methods we like to get the answers we need. For me it's all about working out how to solve the problems and be ready for the when the data arrive. They arrived a few weeks ago and for the past two years I've been slowly building up the software needed to analyse them and to understand the data sets we use. As it happens I was one of the first people to analyse the most recent data, perhaps the first in the world, and seeing the data, using the software I had written myself, is one of the best feelings in the world. It's not something you "do", it's something you create, and becomes your pride and joy. For the past couple of weeks I've been staying up very late and working from home, just to get a few extra hours to analyse the data, and to be the first to see it, because this is a once in a lifetime opportunity. If you want a project that you can make your own, that you can build up from almost nothing into something beautiful and elegant, then data analysis might be perfect for you!

I also develop code in my spare time. When I was a child we didn't have the internet, and all we had at home was a BBC micro computer with 32 Kb of memory, and a handbook to a computer language called BBC BASIC. In those days the floppy disks really were floppy, they were about 13 cm wide, and had to be stored in protective paper sleeves. It was with those tools that I learned how to code and how write simple programs. I knew that this is what I wanted to do with my life (or at least a part of it.) Eventually we got a family computer and a slow dial-up internet connection. Over the years I've learned so many different computing languages, and I keep learning new languages and skills today. Some of my coding projects have been ongoing for over a decade, so it really is a gift that keeps on giving. I think that as well being a lot of fun, coding helps develop a lot problem solving skills, and helps clarify the way we think. It's one thing to ask someone to make a cup of tea, but if you have a tell a computer how to do it then you suddenly realise how precise you have to be! But once you've worked out how to tell a computer how to make one cup of tea, you can tell it to make thousands of cups of tea, you can tell it to tell other computers how to make cups of tea, you can change things around a bit to get a cup of coffee instead. Suddenly all these possibilities open up and you realise that you can do so much more than you could before.

Computing is all about empowerment and extending your reach to do things which are beyond even most intelligent and tireless of people. If there's a problem you want to solve, then the only limits are your own ingenuity and the limits of your hardware. One time I lost my phone and I needed to make an alarm clock, so I spent about 20 minutes making a web page and now I've got an alarm clock I can use any time. Problem solved! I've always enjoyed maths and science and this has motivated a lot of my coding projects. Sometimes someone will tell me about some theorem or a fractal or a pattern, and for the next few days all I want to do is find out more, to analyse it, to find out the answers, and with coding that's possible. If, like me, you have an insatiable urge to discover things and understand things, then learning how to code is one of the best things you can do with your time. If you want to answer a question you can go online, get a little help, find the data you need, and get your answer. Some of the data I've analysed have been general election results, LGBT rights worldwide, stock prices, and how much money I spend on train tickets to visit family and friends (which told me I could afford to visit family and friends more often!)

LGBT rights in Europe, a temporal intersection of politics, social change, and geography, that just begs to be analysed.

Solving problems is one thing, but making games is another. I love to make games, and sharing them with my friends. If solving problems and analysing data are fun and interesting, then that's nothing compared to interacting with other people. Suddenly you have to combine computing with psychology, and that's fascinating. Even more than that you have to keep up with the latest technology, which puts you at the cutting edge. There have been many times where I've looked online for the answer to a problem and found that nobody has posted an answer, so it's possible that I'm the first person to come across it! I love to push technology to its limits and see how far I can push software before it breaks. As soon a a new technology opens up I'm all over it and seeing what can be done, what fun I can have. One of my current projects involves writing educational software about data collection at the LHC. It's something I've wanted to do for a long time, and it's all my own work. As far as I can tell there's nobody else in the world doing anything even similar to what I'm doing, and now I'm getting funding from a UK research council to develop this further and take it into schools. If you have an idea that want to pursue, and the patience to follow it through, then there are so many opportunities that open up for you.

It's not just games though, you can make some beautiful art with software. There's a mathematical fractal known as the Apollonian gasket which is where you pack a space with circles, so I learned how to make this fractal, then decided to use it to fill a photo. Suddenly I had a really cool image that I could print off and put on my wall. Even better than that, I could scale it up to any size without losing any image quality!

Apollonian gasket art.

There is something very satisfying about making something of your own, especially when you show it to a friend and they have a hard time believing that you made such a thing. That's what I do on my website as well, I make an interesting game or a tool and for some reason it doesn't occur to some people that writing your own software is possible. Of course it is, because real people have done it and none of us is particularly special in our talents. It takes a little patience, some practice, and a drive to create something new. As time goes on we find more and more resources at our disposal, more and more experts to help us, better hardware to use, and all this makes it easier than ever to start developing software. There's an excellent community out there as well, other hobbyists who want to share the joy of coding with the latest newcomers. Some of the most fascinating people I've ever met have been other coders, people who freely share their own projects, help strangers out, write games for fun, and write software for the greater good. If you want to see some examples of open and honest altruism then speak to coders online and see what they have made and freely shared. They've made something that has cost them nothing but their time, patience, experience and ingenuity, then shared it with the world. Some things are frivolous, some are crowd-sourced projects that help people out, some lead to revolutionary new techniques or ideas.

It's an exciting time to be a hobbyist or a developer, and I've found that there's no firm boundary between my hobby ends and my work begins. The skills I learn in my spare time I apply to my work, and the problems I solve in my work inspire new projects. I now find myself developing educational software, inspired by my work and initially a hobby, now occupying more of my time and something that will feature prominently on my CV. It's like the saying goes, if you find a job you love you'll never have to work a day in your life!

New data!

Recently the LHC has started Run 2 and this week there are high intensity beams delivering a lot of data. This is a very exciting time to be a physicist and what I have been waiting for the past two years. In fact, my contract as been extended to make the best use of this new data. No amount of working with simulation or theory can compare to working with real data. This job has a lot of advantages and disadvantages, and right now the job is extremely motivating and enjoyable.

What the first 2015 data looked like at CMS.

Something I've wanted to do with new data is automatically analyse them as they arrive, and now I can do that for the first time. (I had tried when I worked on ATLAS, but due to time and technology constraints it was not possible.) Now that we have a globally distributed computing system where any computer can access any file at any time I can stream the latest data each night and run the analysis before I even wake up. So that's what I'm doing right now and it's one of the most fun projects I've ever developed! I've spent the past couple of years preparing and tweaking the software so that I'd have a push-button system in place by now, and it's worked very well so far. There are going to be many teething problems, of course, but they're minor compared to the labour that will be saved. My first large test job is currently running, analysing at its peak 300 events per second (100 events per second, once I/O is taken into account.) Following on from a previous post about how my personal computing resources are not sufficient to do all the work I need to do, I've contacted IT support and been given a generous allowance of disk space, as well being able to run my CPU and I/O intensive jobs overnight. This is going to make a huge difference, and has allowed me to support my students with the simulations and datasets that I analyse. Seeing all this come together is a wonderful experience.

This is how excited physicists get when we first see beams in the machine! I was a very proud Shift Leader that day.

On the one hand this is, of course, very rewarding, but on the other hand a little disappointing that this isn't already being done. There seems to be a culture in particle physics that real work is about manually submitting jobs and analysing data, and there are politics associated with the notion that people should take turns in carrying the burden. We have the technology and expertise to run an analysis automatically, so why not do that? That's what we would see in the private sector. We have access to some of the largest datasets the world has ever seen and some of the most powerful computing resources ever created, it would be a crime not to exploit those as much as possible. I get the impression that I'm the only person doing this kind of work, and if that's the case then it's certainly time to move on to a different workplace. But first, I have data to analyse! I love data.

A perverse design choice

Today I'm spending my time working around a problem that shouldn't exist in the first place. I work on the CMS experiment, using the CMSSW framework, which is usually rather well organised. However it suffers from a problem that is not seen in most other industries, which is that its developers continually and knowingly breaks backwards compatability. This is a major problem because different datasets and simulation campaigns get tied to particular releases, which means that if you develop your own software, as I have done, and you want to use it with different datasets or simulation campaigns, as I do, then you have to take these differences into account.

CMSSW uses C++, so there is no getting around the problem that the method signatures have to match exactly. As an example I have the following lines of code in CMSSW_5_3_11, which is used for the 8 TeV data:

  beamSpotLabel_ = iConfig.getParameter("beamSpot") ;

For later CMSSW releases the code looks like this:

  beamSpotLabel_ = consumes(iConfig.getParameter("beamSpot")) ;

As you can see the implementation has changed in such way that if I want to use the beamspot in CMSSW_5_3_11 and later versions I need to carry around two sets of code. Unfortunately this isn't the end of the story because I have five different CMSSW releases in parallel, with different changes in implementation between each one, so I need to keep track of five different sets of tweaks just to get things working. For a while I had five different branches and instructions on which ones people should use, but this meant making five pull requests and five merges each time, because merging to a master branch would overwrite all these small tweaks when applying it to the separate branches. As a result I would end up with five branches diverging by more and more each time someone makes a pull request, and would have to maintain them manually, and all the while GitHub would be complaining that things aren't up to date.

The way I got around this was to use C++ comments and use a script to change the CMSSW release. I have to admit that I find this solution to be quite clever. The above code would be replaced with something like this:

// CHOOSE_RELEASE_START CMSSW_7_0_6_patch1 CMSSW_7_3_0 CMSSW_7_2_0 CMSSW_6_2_5 CMSSW_6_2_0_SLHC23_patch1
  beamSpotLabel_ = consumes(iConfig.getParameter("beamSpot")) ;
// CHOOSE_RELEASE_END CMSSW_7_0_6_patch1 CMSSW_7_3_0 CMSSW_7_2_0 CMSSW_6_2_5 CMSSW_6_2_0_SLHC23_patch1  
/* CHOOSE_RELEASE_START CMSSW_5_3_11
  beamSpotLabel_ = iConfig.getParameter("beamSpot") ;
CHOOSE_RELEASE_END CMSSW_5_3_11   */

Then all I need is a python script to go though and comment/uncomment the relevant parts to match the release. This way I can keep one single branch with all the pull requests and keep everything up to date with minimal fuss. When I told a friend that this is what I was doing she was shocked. What sort of organisation would break backwards comptability so regularly, especially if it means that code that works at 13 TeV won't work at 8 TeV? It turns out the answer is the CMS experiment would do that, and this makes code development time consuming, tedious, and potentially dangerous. I don't want to be around in a few years time when most people can't remember how to use CMSSW_5_3_11 for the 8 TeV data.

I thought things were under control, except now there is a new problem. Suppose a developer adds a new feature and submits a pull request- which CMSSW release are they using? Is it important to specify which CMSSW release they use? So now I have to write yet another script that goes through all five releases, create environments for each one in turn, copy across the source code and attempt to build it. Every time there's a new feature, no matter how minor, it has to be tested with all these different CMSSW releases. It also has to be tested in a "safe" space where other CMSSW relases can't interfere with it. Suppose I am developing in CMSSW_7_3_0, which is one of the most recent releases. I create a new release area, check out the code, do my development and test it. Now I need to test it on the other four releases, and do so somewhere outside the current release area. That means going up through the directory structure, making five new release areas, copying the source code, setting up the environment, doing build clean, then build, and fixing errors as they arise. This is not at all pretty, and it's also a bit of an imposition on the user, since it makes temporary directories in their user space. However that's the best I can do under the circumstances and we'll see soon enough if it's sufficient. This all assumes that the user is not using a sixth CMSSW release, which is something else I need to take into account...

CMS has made a perverse design choice that is taking up a lot of my time that could be better spent on actual physics analysis. I think that's an excellent reason to move on to a more sane coding environment somewhere in the private sector.

Edit: After speaking with some colleagues (who sympathised) and friends (who gave some useful advice) I realised that the "correct" way to handle this situation is to use compiler directives. I have other people using the software, so for now I'll keep the current solution, but when a suitable time comes I'll consider using compiler directives to manage the build. However, even that brings its own challanges.

A loss of life and talent

Two weeks ago one of my close friends, Moritz, died in a climbing accident. I've blogged about it a lot on my other blog, and it's had a profound effect on my attitude to work. The decision to study particle physics followed my brother's suicide, when I needed time to myself and needed to put off big decisions about my life. I've been prioritising my professional life over my personal life ever since, but now my needs have changed. I spent a long weekend in the UK with some good friends who know what it's like to grieve, and some who knew Moritz. In contrast to that, life in Brussels and at CERN seems cold and I've lost a lot of enthusiasm for physics. I've been keeping this blog for over a year at this point and for over a year I've been planning my next steps. This has been accelerated by losing Moritz. For the past year I've used my spare time to develop plans for the future. With Moritz's death my attention has suddenly been focused more on the future and find myself building up the infrastructure I need to move back to the UK with some startup companies under my belt. For the past few days I've found myself contemplating moving back as early as August, before we get the most interesting LHC data. At this moment my heart isn't in physics at the moment, and it feels like my enthusiasm for it died with Moritz. It's made me feel very strongly that CERN is in the past and even though I didn't see much of a future for me in particle physics research, at the moment I'm struggling to see how it's relevant to the present day. I don't know if this will be a temporary feeling or not, so I'm waiting before I act on it, and I might get my passion back for one last hurrah before I move on to more exciting and challenging things.

Moritz as I remember him, full of ambition and on top of the world.

Moritz was incredibly bright and ambitious. Particle physics has lost one if its most promising minds, just as his career was about to take off and go to new and exciting places. His loss to field is one more piled on top of so many promising young people who have failed to find good jobs, in spite of their brilliance and genius. The field is harsh and cruel, and although it's just bad luck that Moritz died so young it's one loss too many for me. I don't want to spend any more time in a field which seems to repeatedly ignore talent and ingenuity (which it also did to Moritz) while there are other opportunities out there just waiting to be taken.

I don't want to make rash decisions based on fear, or regret, or grief, or frustration, which is why I'm waiting before I decide when to leave. There's also no reason to think that another choice won't have all the same problems. But given the choice between living in yet another foreign country for relatively poor pay and poor working conditions, and working close to the people who actually care for me when I need them the most it's an easy choice to make. I'm done giving particle physics the benefit of the doubt and at this point only a UK based fellowship where I got to set my own agenda would keep me in the field.

(This post was written en route to Moritz's funeral.)