React: overkill with Redux and where Portals save the day

Recently I encountered a situation where I needed to overlay a div on top of another div. The standard way to do this is to use CSS and absolute or fixed positioning (https://stackoverflow.com/questions/2941189/how-to-overlay-one-div-over-another-div).

Sounds simple enough, however I need to do this from within a deep hierarchy of elements:

-> Detail Component
  -> Action Bar Component
  -> Items Component: Loop through and display expandable content divs
       -> On Click : overlay a component which is a div containing "expandable content"

The "items" component uses a standard bootstrap table to align the elements like so:

item 1	item 2
item 3	item 4

This layout means that when you select the onClick() for "item 1" and you render the expandable component which is essentially just a div, and it is positioned relatively the the content will appear in place of item one in the table.

item 1 (expandable content)	item 2
item 3	item 4

Cut a long story short, every which way I tried to style the popup unwanted styles would be inherited from the table and other components above it. I also found that I had to place it absolutely to get the effect I was looking for which created a raft of further problems. Finally, it was suggested that this might be a good use case for React Portals which did indeed save the day.

I also learned on this project how much Redux can be over utilized. "When you have a hammer, everything starts to look like a nail". This was said a lot, and it should be, because when you are deep into the detail of a project it can be hard to see that it has been over-engineered.

Isomorphic or Universal Javascript

Why I have decided to learn about this?
I first truly took note of the term "Isomorphic JavaScript" when a recruiter sent me through a job which I thought looked extremely interesting. At the time I was not looking for a job, but it reminded me about the long standing problem with decent SEO and Single Page Applications (SPAs). I had heard a lot about React and Angular 2 providing some way to solve the problem, so I was curious.

Notably, the company in question were listing some other design patterns and technologies which were new to me and were also of interest: 12 factor app, Mesos and Marathon, CircleCI.

Godammit man, what does Isomorphic mean!
Isomorphic means "corresponding or similar in form". So essentially we are describing some sort of server side rendering to support the client side application.

Or as the AngularJS Universal Repository puts it: "A JavaScript Application that runs in more environments than just the browser".

The advantages?

• The application does not need to rely on JavaScript being turned on in the browser for it to function
• The application does not need to have a loading gif to explain to the user that patience is required
• THE BIG ONE - The application can be more easily crawled by search engines because the page is also available from a static URL.

Some disadvantages and limitations

• "Uncanny valley" - this is a term used to describe the time between the application appearing to be functioning and available and the time before it is actually responding to the users demands. This can be confusing for the user.
• Loss of separation between back-end and front-end application code. However, I would say, in this API/ Micro-service architecture world, that this actually makes quite a bit of sense. The User Interface will still be separated from Business Logic, we are just blurring the line between server-side "Get" request responses and view rendering.
• This is still just a bootstrapping technique and some SEO issues do still remain.

For a more thorough discussion on why Isomorphic JavaScript is "not the answer" take a look at this discussion on ycombinator (at your own risk of course!).

The dev-ops cycle (a tale from the trenches)

Hi, I would like to describe to you a problem which caused real pain for me and the company I work for. And then I would like to explain to you why these problems were occurring and how we kept failing to address them. And finally what was improved as a result.

Some Background (Windows Workflow)

At company x we have been using Microsoft's Windows Workflow Foundation hosted and scaled using a number of Azure Web Instances. In other words, when there is a problem with the workflows it adversely effects the end users. Pretty flawed design, but this is what we had been working with for sometime.

In general Windows Workflow is a highly effective way to manage software development. It offers a clear overview of a highly complex systems and a highly scalable run-time engine out of the box. This guy (Blake Helms) is a major fan and so is the CTO at my company.

Of course, Windows Workflow does have some peculiarities. See these excellent posts on the dreaded tight loop and managing the workflow persistence store.

A Case Study

On Thursday afternoon our web instances started bouncing wildly between 50% -100% CPU usage, our users were complaining that the site was "slow", and restarting one web instance at a time was only solving the problem temporarily. Looking through the logs it seemed that this had been going for a few days to a lesser or greater extent depending on load. The same pattern was occurring again and again each day getting worse and more noticeable - the overall CPU usage would ramp up and then ramp back down again temporarily disabling Web Instances.

Having spent some time looking at this with our normal analysis techniques it seemed that there was no problem with the workflows. Looking at the workflows coming in every couple of minutes showed that they were being processed as expected. I suggested that we see what happens when the users log off at the end of the day - hoping that we would see a clearer picture out of hours.

Accept The Problem

Dev-ops problems tend to be unpopular in organisations. Most people dislike being disturbed of course, but it is also the stress and fraught nature of such incidents which cause Developers to shy away.

I personally believe that there are huge incentives to take on real live problems, but to reap the benefits they must be seen through from start to finish. Like many things in computer science, when a problem is hard do more of it! This, I guess, is why we are seeing more developers become dev-op specialists.

The Danger

The danger with all dev-ops is that we spot the problems and find a work-around, because this gets the situation off our backs.

This behaviour can create the "dev-ops infinite loop" because the real source of the problem isn't fed back into the development process.

As I am writing this now, it sounds obvious: find the issue and make sure there is a pull request into the next release which fixes it, what's the problem?

Well, it is quite possible that you may not have considered where or how the situation came about from the very beginning. In larger organisations a particular release may have involved many people, which could mean that vital information is lost or unavailable to you. It could also be that information may have been forgotten completely due to a release cycle that is too long or too complex.

Understanding Why - Continuing The Case Study

By Friday team was starting to get desperate because the problem was not going away and we still could not understand why the problem was happening. There seemed to be no difference between out of ours and in hours. A few theories were put forward with minor bug fixes in the code, one of which was a bug which had been present in the system since nearly the first release, lets call this "Theory 1":

• Email templates are being compiled through the Razor engine every time an email is sent. This causes the CPU usage on the Web Instance to ramp up because it is such a resource heavy operation.

If you have worked with Razor in a highly scaled system, you may have also come across this problem. A fix was developed and released over the weekend. The email Razor templates would now be pre-compiled on app startup thus reducing the load and solving the problem.

However, come Monday morning this did not calm the server instances down. Every time we ran up a new server instance the load would be OK for a time but then would start bouncing up and down giving our users a poor experience when using the site.

A Moment Of Monday Clarity

"Theory 2":

• There is a workflow which is running which is blocking the web instance and causing the increase in CPU usage.

We had to accept on Monday that it was back to the drawing board. The workflows had been disregarded as a potential cause because the monitoring tools we had written to check the database showed that the workflows were being processed very efficiently. However, there seemed to be no other plausible explanation.

Further analysis on the "workflow instance table" in the database showed us that there were a growing number of workflows being fired of a particular type at very particular intervals. We were able to identify this by changing our workflow monitoring tools to look at the number of workflows queued of a particular type, rather than how many workflows were suspended or overdue.

The Temporary Fix

On Monday evening we were able to push a small fix which restricted the workflow in question so that it ran only in out of office hours. We hoped this would get the users and the CEO off our backs - yes it got pretty serious! But we were still not entirely clear on what the route cause of the problem was.

Finding A Lasting Solution

The final solution to the problem was eventually deduced through some more thorough detective work. The kind of detective work that is hard to do properly when under pressure.

What went into the last release that could be very resource hungry? And what could cause this hunger to increase gradually over the days that proceeded afterwards?

The answer turned out to be from a new more generic workflow which had been written to replace a number of other workflow processes. There was actually nothing wrong with what the workflow was doing. The only error that the developer had made was to fail to consider its impact when deployed "en masse" if you will. This workflow had been written by a developer who had left the company some months before. Something like this would have been tricky to predict.

Areas For Improvement

I would say that this tale lays bare some quite common problems within software organisations.

• Before the software was released, there was no effort made to load test the solution.
• When the release was promoted to production, there were so many changes made by so many developers over so much time that there was no "scrum master" or technical leader who knew enough about what was in the release.
• After the release was running, the tools for monitoring the solution where not sensitive or sophisticated enough to show a larger than expected increase in server resource usage.
• The problem went unnoticed for too long.
• When the problem was identified, developers hoped that it would just "go away" without pursuing the problem vigorously enough.
• Developers put too much faith in one solution working without thinking enough about developing some ideas on other solutions.
• Nobody in the team was calm enough to look through the new feature list and suggest a feature with risks.

What has happened since this incident:

• UAT goes through a load test before being promoted to Production.
• Releases are kept much smaller and more frequent
• The workflow database monitoring tools have been improved
• We gave ourselves a kick and tried to learn as much as we can from the case.
• The incident resolution process has been looked at.

Things that I wish I had known when I started out in the industry

I think there are some interesting things which I have learnt from my time as a developer. Some of them are nothing to do with code and some are very general but anyway, in no particular order this is my brief survival guide:

• Ask, ask, ask - until everyone is sick of you, until your manager wants to gauge out you eyes with a blunt spoon. Its the only way. Especially when you join a company, sure they want you to contribute... but mostly they want you to learn without destroying their clients confidence in the software! The bottom line is: managers actually prefer to be hassled all the time even if they don't realise it yet!
• Always, especially when you are very confident in code, add in try catches in appropriate places so that errors can be easily traced by you or other members of the team. In all fairness, I often find myself getting lazy with this sort of thing from day to day. When I have been away from back-end code and have had my head in CSS for sometime or I am deep in a large re-factor. But seriously, don't make your life difficult when code is released into the wild! Code which is silently failing is much more dangerous than code which is obviously broken.
• When writing complicated business logic always unit test. It will save your bacon later down the line. When you find a bug, write a new unit test.
• When writing unit tests always test from the lowest possible level up. If you are mocking loads and loads of objects, your code is probably too tightly coupled.
• Communication with other team members. Lets face it we all say things on our CV like "I am a great communicator" and "I won the first prize in spelling when I was 12", OK less the second one, but the point is, we all think we can talk to other people when necessary and get our point across. However, what about all the extra things? such as an estimation on time or a report on a technical challenge that was overcome? As a major technical resource for your company, they need you to speak up on what is going on. Sure, you should have a manager that does this for you, but why not help him out a bit and do it yourself?

Taking the plunge and starting a technology blog

I have finally got around to writing a blog, woop go me. I have joined the thousands of other technology geeks blogging about stuff that only other technology geeks care about. To be honest as a web developer in my professional life I can't help feeling a little grumpy when it comes to this kind of shameless self promotion.

However I am going to take the plunge anyway, in truth I am mostly going to use this blog to shame myself into writing more code. We can all learn from Jennifer Dewalt.