food for thought: code bloat

[Zoli]

My point of view is a little different: when itunes come out, the music industry business model already was under pressure from the digital formats(and not only P2P); without Apple opportunistic grab, the music situation now would be similar of what is now on the video streaming market. And to remind you about how customer friendly is Apple: DRM had been removed when Amazon started to sell MP3's, lossless option had been added when Bandcamp(IIRC) started doing. Presenting Apple as customer focused company is hypocrisy.

Amazon started selling DRM-free MP3s from EMI and Universal on 25/9/2007.

Apple started selling DRM-free AAC songs from EMI on 10/9/2007.

Apple was first, and it was the labels that were the hold-up, not Apple. Apple had been doing the hard work trying to persuafe them to go DRM-free for years.

Apple had the distribution in place for years; when the Amazon deal came thru, Amazon still had to build up the distribution. Apple had to flip a switch, once the Amazon deal was sealed, that's how he managed to beat Amazon with two weeks. But the most important part of the deal(from the labels POV) wasn't the DRM-free distribution, but the multiple storefronts.
As conclusion to our discussion, care to comment of the original DRM terms, especially the one which locks you in the Apple ecosystem(unlimited copies on Apple devices)?

[Berni]

Apple did the most important part of the work in all this. Getting the legal nitty gritty sorted so that the record labels would allow them to sell music using there 'weird new' song by song model. Said record labels are the same people who wanted to instantiate a tax on recordable media (with the excuse to recoop piracy losses), pushed for embedding copy protection data into all digital audio interconnect standards, or placing rootkit viruses on CDs to keep the music on it locked down if inserted in a PC (but also bluescreen computers sometimes)

To make this deal happen the record labels likely pushed HEAVILY for all sorts of DRM to be implemented, the big bosses there must have been absolutely terrified of Apple just handing out MP3s of there music. So Apple would of course offer to implement forms of DRM that fit well into there vision of what the iTunes Store should be. Given that Apple is about building this ecosystem where all apple devices work seamlessly together means that this is a form of DRM they are on board with.

Then after the iTunes store has taken off like a rocket the big bosses at the record labels saw this "stupid way of selling music" as actually beating there own "tried and tested physical CD sales model" so actually started supporting it. The record labels are a very stubborn bunch, so this was a massive effort on Apples side to force them into taking the first step towards a new way of music distribution.

Of course Apple didn't do this because they wanted music to be more accessible to people. The reason they did this is to differentiate there iPod product from the rest of the MP3 players. You no longer had to be a pirate to obtain MP3 files of music you legit own for use on your MP3 player. If you bought a iPod then all you needed to do is open iTunes and buy any song you want for 1$ then wait a few minutes for the songs to be magically transferred to your iPod device. This makes it very easy (even for non technical users) and convenient to fill up your MP3 player with perfectly legal music. So yes Apple did it for there own profit, but in doing so set the stage for internet music distribution.

In a similar way Steam provides a DRM mechanism for its PC games where these games will only run using the Steam client. But it is so convenient and  unobtrusive that most people actually prefer to buy games on Steam rather than say GOG.com, where you get the game DRM free (as simply a giant zip file that you have to then store on your harddrive for the lifetime of owning it). The Steam DRM is also really easy to defeat by simply replacing one steam DLL with one that simply claims you own all games. It is that simple on purpose because they don't want DRM accidentally locking people out of a product they bought. However Steam still allows game publishers to add in any extra DRM they like, but it is there own fault if there own strict DRM turns away buyers.

[Siwastaja]

In a very real sense, convenience is a root cause for code bloat, too.

Indeed. Throwing in a library or doing it the dirty inefficient way is usually easier so that's what people do.

Yet, instead of convenience, I would say, impression of convenience.

Because too many times you see this pattern that doing X from scratch would be 1-day job or whatever, but you can't be allowed to do that, because it is "not convenient", it is "too tedious", and you need to use "time-saving" strategies like managing library dependencies, learn the library, write wrappers, and whatnot, so that 1-day job becomes 1-month job or even worse, when it is about bloated frameworks instead of small libraries, 1-year job.

Then management people assume that hey, by using this trend framework of the year, which is the best thing since sliced bread (this is universal law of physics, like the speed of light, and it can't be questioned), it took a full man-year to get X done, so "imagine how difficult it would have been without the framework, you would have had to write all this 57 000 000 LoC from scratch, isn't it convenient this is all available to us!"

In other words, I don't prefer, but I can accept the priorization of short-term over long-term, i.e., to ship something quickly. But this is not the trend I'm seeing. In reality, we see fairly simple projects get overly bloated, way over budget, way over schedule, supposedly developed using strategies that make things "quick", but result being the opposite: only the quality matches the "fast&ugly" expectation, but cost is cathedral-level.

Hence, I believe the first step towards rectification of this bullshit situation would be to truly measure the time and money poured into software projects, and start doing that buggy unmaintainable crap quickly and cheaply, because now we have buggy unmaintainable crap done slowly and expensively. I.e., don't try to fix quality first, fix the cost side first.

After we trim all the fat off and get into minimum viable prototypes, we can start building sustainable long-term practices, i.e., improve the quality of the product making short-term sacrifices regarding time and money.

But right now, it's impossible to ask for more time and money, because software projects already are way too expensive.

[brucehoult]

As conclusion to our discussion, care to comment of the original DRM terms, especially the one which locks you in the Apple ecosystem(unlimited copies on Apple devices)?

The record labels were convinced by Apple's security mechanisms, but not by others?

Once they dropped DRM that wasn't important of course.

It was Apple pushing the dropping of DRM.

[nigelwright7557]

Bloat exists because it's currently cheaper to leave piles of shit everywhere than clean it up.

When performance or user experience suffers in any way then it will change.

There's a massive performance and energy usage wall coming up on the hardware side of things that will put the focus back on software efficiency.

With 5GHz processors and many gigs of DRAM it doesnt matter as much these days.
In the distant past I remember scraping the last byte out of assembly language programs to use less memory and gain more speed.

[Berni]

With 5GHz processors and many gigs of DRAM it doesnt matter as much these days.
In the distant past I remember scraping the last byte out of assembly language programs to use less memory and gain more speed.

It does if you have to wait >30 seconds for a TV to boot up. Or when i open up excel and add a graph of a table with 50k points and the whole program locks up for 10 seconds before the graph suddenly pops in.

Sure we might not have to think about saving every last byte these days, we got plenty to throw around. But at least optimize your software to the point that it feels like it is running on a 5GHz machine rather than on a Pentium II

[nigelwright7557]

This is a problem in all forms of writing.  There is a famous quote attributed variously to Blaise Pascal, Mark Twain and Jane Austen among others.

"I don't have time to write you a short letter, so I am writing a long one."

It takes time, energy and thought to reduce code size.  All of these are of limited availability in development.

A lot of people still pile in and start writing code instead of planning it.
If you fail to plan, you plan to fail.

[Nominal Animal]

A lot of people still pile in and start writing code instead of planning it.
If you fail to plan, you plan to fail.

True.

Although, I do tend to write code before I finalize a plan: I create separate test cases to check the key parts and algorithms of the core logic, to make sure the plan I'm planning on is on a solid ground.  I usually learn a lot about it in the progress, and end up rewriting the part anyway in the actual project, so one could consider it wasted effort, but I don't mind: understanding the key parts of the algorithm is worth the time and effort spent to me.

Example:

In certain types of molecular dynamic simulations, temperature control is implemented by simply scaling particle velocities.  This works well for bulk, but for small independent clusters, the initial random velocities do not cancel out perfectly, and the temperature control just makes them rotate faster.  In some cases this is okay, in others it is unwanted.  So, one solution is to determine the particles that form such clusters, and dampen their rotation.

How exactly do you determine which atoms are part of a cluster, when you have some kind of gas permeating the simulation volume?  You might use rules of thumb, but they'll likely fail.  If the simulation is about growing clusters in a very low density hot gas (a very common real life case), atoms are constantly aggregating into the cluster.  In fact, you don't even know how many clusters you have in the simulation.  Asking the human to select the atoms in the cluster is not viable, since simulations tend to be run on HPC clusters in queues, not interactively.

It turns out there is a trivial solution that costs very little.  It is based on the disjoint-set data structure.  Basically, before each time step, you initialize the disjoint set (with an unique integer for each atom, basically numbering them).  Then, when you calculate pairwise interactions –– these simulations almost always use classical potential models, not quantum mechanical ones –– you merge any sets that are within a cut-off distance, approximating covalent and ionic bonds.  (You can do a more precise rule, based on potential energies and the force between the pair of atoms, but it turns out to not be necessary for this.)
At the end of the time step, you flatten the disjoint set paths, and you end up with a cluster identifier for each individual atom.

To implement this, you wouldn't just start modifying your favourite simulator like Gromacs or LAMMPS –– both are open source, so you're definitely allowed to.  Or, well, you'd start by experimenting and testing, before formulating a plan.

Without a plan, just making changes that seem to produce the results you want, you're risking the results of all simulations done by the modified version!

After doing the initial tests so you roughly know where to add or modify the stuff needed, you write a plan.  Then, you check the plan against the logic of the existing simulator, perhaps talk to the developers on the mailing list for confirmation, and only *then* do you start implementing the actual changes.

Now, this may sound like a lot of work.  Thing is, it actually saves total effort.  (At least if we count in the efforts spent to debug willy-nilly made changes and the ensuing erroneous results and other users' efforts in finding why.)

I do not treat scientific simulations any differently than I do normal applications.  I do not want my applications to silently garble or lose data; at minimum, I want them to tell me whenever they detected something unexpected.  Having them do retries and workarounds is a plus.  None of this is really hard; it just takes developers that care.

[SiliconWizard]

This is a problem in all forms of writing.  There is a famous quote attributed variously to Blaise Pascal, Mark Twain and Jane Austen among others.

"I don't have time to write you a short letter, so I am writing a long one."

It takes time, energy and thought to reduce code size.  All of these are of limited availability in development.

A lot of people still pile in and start writing code instead of planning it.
If you fail to plan, you plan to fail.

Yep. But the "trick" here is to clearly define what "planning" means in a given context.

If, as is very common these days, "planning" is almost only about timing - focusing on when things must be done - this is going to please top management (until they realize that was all bullshit anyway and that tech plannings rarely hold), but it's not going to help much otherwise.

Now if by planning you mean "architecturing", then sure. Sadly, this tends to be a forgotten notion in software these days, and if you talk about software architecture, you're likely to be considered a dinosaur stuck in the waterfall days. Yeah. And, yeah, people tend to focus on short-term rewards. Just like in other areas anyway. This isn't just with software development.

But anyway. The current state of software development makes me pretty sad overall. Unless you have full control over what you do and how you do it, I would recommend not walking, but running away from software development. Just do something else. Really. And leave it to the ones who are happily churning along.

[Kjelt]

Now if by planning you mean "architecturing", then sure. Sadly, this tends to be a forgotten notion in software these days, and if you talk about software architecture, you're likely to be considered a dinosaur stuck in the waterfall days.

The problem is that the term SW and System architect are watered down to mineral water these days.
I now see architects with less than 5 yrs of work experience f*cking it up because they have no clue what they do except look good and talk the talk to pl and mgt in the ten meetings a day they are sitting in.
I said it before in the 90s the career track was
jr sw eng, mr sw eng, sr sw eng, jr sw arch, sr sw arch, jr system arch, sr system arch.
And between each step there would at least be 3 to 5 yrs experience.

But indeed it is all watered down, having sw engs from a different background like biology or history , they do not know design patterns, the solid principles. If or unfortunately when they become architect in a few years just because they hold an academic grade (in biology) you never know what is going to happen 

[PlainName]

That's old thinking. Nowadays you architect a product by using Python to glue github modules together. Easy peasy.

[brucehoult]

That's old thinking. Nowadays you architect a product by using Python JavaScript to glue github node modules together. Easy peasy.

[SiliconWizard]

That's old thinking. Nowadays you architect a product by using Python to glue github modules together. Easy peasy.

 

[Bud]

System architect are watered down to mineral water these days.
I now see architects with less than 5 yrs of work experience f*cking it up because they have no clue what they do except look good

Do not know what your justification fot 5 years experience comes from.... Many years ago I had 2 years of Extensive experience working in a specific area and I still carry on decades after and get hired and get paid for the knowledge I got from that time period. Granted, I did not sit twiddling my thumbs during those 2 years, but I firmly believe 2 years is a huge experience absorbing resource. Even 1 year, because in my second year new knowledge influx dropped exponentially - simply there was nothing new to learn.

[perdrix]

I don't know. I'm next to (I refuse to say I'm with) a team of 300 developers and quite frankly they have enough time to do a proper job of solving all the problems. They just lack the motivation and skill to do so.

I saw them burn 9340 hours on something that took me 30 minutes to fix. No shit. That one was fucking buried as well so the management didn't look like morons.

Been there seen that

D.

[SiliconWizard]

I don't know. I'm next to (I refuse to say I'm with) a team of 300 developers and quite frankly they have enough time to do a proper job of solving all the problems. They just lack the motivation and skill to do so.

I saw them burn 9340 hours on something that took me 30 minutes to fix. No shit. That one was fucking buried as well so the management didn't look like morons.

Been there seen that

Indeed.

[Kjelt]

Do not know what your justification fot 5 years experience comes from.... Many years ago I had 2 years of Extensive experience working in a specific area and I still carry on decades after and get hired and get paid for the knowledge I got from that time period. Granted, I did not sit twiddling my thumbs during those 2 years, but I firmly believe 2 years is a huge experience absorbing resource. Even 1 year, because in my second year new knowledge influx dropped exponentially - simply there was nothing new to learn.

Always exceptions and it also greatly varies with the domain and scope you are working on.
Surely you agree understand that the software architecture for lets say an thermostate is more limited than for a tv which is again more limited then for an electron microscope etc.
At my current company a new sw eng will only be productive eg earning money for the company understanding what the work is after two years. My previous company that was two months.

But to answer your question, the 3-5 years was the default HR evaluation period with a local large electronics company in the 80s till 2000s. Nowadays it is more dynamic, eg there are brilliant people that stand out and get a fast track.

[IDEngineer]

Sometimes code bloat IS the most efficient solution. "Good, fast, cheap: Pick any two." It totally depends upon the problem being solved. I bet the Apollo 13 astronauts didn't want to delay their return to wait for code jockies to slash a few cycles. On the other hand, they cared very much about optimizing current consumption because their fuel cells were gone.

Not every problem is a PhD thesis. Sometimes "good enough" is exactly the right answer.

[Siwastaja]

Sometimes code bloat IS the most efficient solution. "Good, fast, cheap: Pick any two." It totally depends upon the problem being solved. I bet the Apollo 13 astronauts didn't want to delay their return to wait for code jockies to slash a few cycles. On the other hand, they cared very much about optimizing current consumption because their fuel cells were gone.

Except that bloated shit usually is expensive, delivered late, and does not work, while efficient software is also delivered in time, cheaper, and works better. The correlation is strong.

This is because poor practices go hand-in-hand. If you can't write decent software, or manage a software project properly, then it will fail in many different ways.

Shaving off the last 1% of performance is of course totally different, but it's never about this really.

[Berni]

Sometimes code bloat IS the most efficient solution. "Good, fast, cheap: Pick any two." It totally depends upon the problem being solved. I bet the Apollo 13 astronauts didn't want to delay their return to wait for code jockies to slash a few cycles. On the other hand, they cared very much about optimizing current consumption because their fuel cells were gone.

Not every problem is a PhD thesis. Sometimes "good enough" is exactly the right answer.

The code in the apollo computers did pick compromises where appropriate.

A lot of the firmware is in the form of bytecode executed using an interpreter to save program space while some sections of code are executed as raw machine code in order to run faster. The computers they had ware not very fast, so speed optimized code was needed in some cases. These early computers also didn't have power saving states to drop into while running.

Optimizing too much in one direction is often a bad thing.I am not asking for people to optimize every last cycle of speed out of there code. Just do the basics to catch the worst speed bumps or resource hogs in code. Picking a slight shortcut that runs at 10% the original speed is not excusable by "It still runs fast enough on my 11th gen i7 machine"

[brucehoult]

The code in the apollo computers did pick compromises where appropriate.

A lot of the firmware is in the form of bytecode executed using an interpreter to save program space while some sections of code are executed as raw machine code in order to run faster. The computers they had ware not very fast, so speed optimized code was needed in some cases.

An obsolete concept.

Once you have registers the same size as the data you are manipulating (including pointers), and enough of them, a well-designed RISC ISA with nearly direct hardware control (i.e. simple decode, no microcode) implemented in the now classic 5 stage pipeline is both as fast as simple hardware can go AND as compact code as any interpreted bytecode.

There is no longer any need to choose between size and speed.

[Berni]

Yeah it doesn't make sense to do this approach on modern machines. Even just ARMs Thumb instruction set is reasonably compact and flash commonly comes in hundreds of KB these days.

Most of the reason this interpreter approach made a lot of sense on the Apollo guidance computer is that the instruction set on it was very limited (really couldn't do a whole lot per instruction in the couple of instructions it had). At the same time program memory was a lot more expensive when it had to be woven trough ferrite rings by hand and actually increased the total mass of the computer. That interpreter must have been a pretty fancy bit of software engineering back in the day.